Observation of behavioural skills by medical simulation facilitators: a cross-sectional analysis of self-reported importance, difficulties, observation strategies and expertise development

Abstract

Background

The association between team performance and patient care was an immense boost for team-based education in health care. Behavioural skills are an important focus in these sessions, often provided via a mannikin-based immersive simulation experience in a (near) authentic setting. Observation of these skills by the facilitator(s) is paramount for facilitated feedback with the team. Despite the acknowledgement that trained facilitators are important for optimal learning, insight into this observation process by facilitators is limited.

Objectives

What are the self-reported current practices and difficulties regarding the observation of behavioural skills amongst facilitators during team training and how have they been trained to observe behavioural skills?

Methods

This cross-sectional study used a pilot-tested, content-validated, multi-linguistic online survey within Europe, distributed through a non-discriminative snowball sampling method. Inclusion was limited to facilitators observing behavioural skills within a medical team setting.

Results

A total of 175 persons filled in the questionnaire. All aspects of behavioural skill were perceived as very important to observe. The self-perceived difficulty of the behavioural skill aspects ranged from slightly to moderately difficult. Qualitative analysis revealed three major themes elaborating on this perceived difficulty: (1) not everything can be observed, (2) not everything is observed and (3) interpretation of observed behavioural skills is difficult. Additionally, the number of team members health care facilitators have to observe, outnumbers their self-reported maximum. Strategies and tools used to facilitate their observation were a blank notepad, co-observers and predefined learning goals. The majority of facilitators acquired observational skills through self-study and personal experience and/or observing peers. Co-observation with either peers or experts was regarded as most learn some for their expertise development. Overall, participants perceived themselves as moderately competent in the observation of behavioural skills during team training.

Conclusions

Observation of behavioural skills by facilitators in health care remains a complex and challenging task. Facilitators’ limitations with respect to attention, focus and (in)ability to perform concomitant tasks, need to be acknowledged. Although strategies and tools can help to facilitate the observation process, they all have their limitations and are used in different ways.

Background

The global interest in so-called behavioural skills increased significantly over the past decades. This was especially due to the association between behavioural skills and preventable medical errors, the latter resulting in unnecessary patient morbidity and mortality [1,2,3,4,5].

Behavioural skills or non-technical skills (NTS) can be defined as the cognitive, social and personal skills that complement ‘technical skills’ and contribute to safe and efficient task performance [6]. Cognitive NTS is often subdivided into situation awareness, decision-making, coping with stress and managing fatigue, whereas social NTS consists of task management, leadership, communication and teamwork [7, 8]. Related terms for NTS commonly found in literature are ‘crew’ or ‘crisis resource management’ (CRM), risk management, ‘soft skills’ and ‘crisis avoidance resource management’. Murphy et al. [9] proposed the use of ‘behavioural skills’ as a new lexicon over the term NTS as more accurately describing the range and complexity of interpersonal skills.

With the acknowledgement of behavioural skills’ importance in the prevention of medical errors, many team training programs focussing on these aspects were developed. Currently, well-established literature is available which underscores the importance of team-based behavioural skills training [10, 11] and evidence correlates these interventions to improved team performance and even improved patient outcomes and reduced mortality [12,13,14,15,16,17].

Commonly, these team-based trainings are offered through ‘immersive simulation’, during which teams are confronted with a scenario-based mannikin in a setting as authentic as possible. Feedback is usually facilitated in the debriefing phase by the facilitator. This debriefing phase is paramount in simulation-based training as this is where most learning occurs [18,19,20]. Savoldelli et al. [21] found that simulation training without feedback did not lead to improvement of behavioural skills in anaesthesia trainees. Furthermore, a meta-analysis showed properly facilitated debriefings to be three times more effective compared to non-facilitated debriefings [22]. Additionally, self-assessment of behavioural skills by participants is reported to be difficult, underscoring the importance of trained facilitators to guide this paramount process of debriefing [22,23,24].

Roussin and Weinstock [25] described a classification for the broad range of simulation-based training programmes currently available. The so-called zone 3 and 4 training entities encompasses ‘native team’ training (i.e. without role-playing), with a focus on behavioural skills and a facilitator (rather than an instructor) to guide the (post-event) debriefing. This is the type of training meant wherever this article refers to as ‘team training’.

Problem definition

The process of ‘facilitated feedback’ during debriefing encompasses different stages: first of all, facilitators have to make observations of the behavioural skills. Then, they have to interpret and evaluate these observed aspects before they can facilitate a feedback conversation with the team. Despite the acknowledgement that trained faculty is paramount for optimal learning, insight into how facilitators observe behavioural skills and how their training can impact behavioural skill observation, and therefore, participants’ learning is still limited [26]. The available literature has mainly focused on ‘standardising’ this observation process at the end, by providing facilitators with tools, e.g. ‘behaviourally anchored rating scales’, to assess team performance [27]. This however does not describe what facilitators find important to observe, nor their reasoning and interpretation. More so, observation and provision of feedback in behavioural skills can be regarded as deceptively ‘simple’ due to the availability of many easy-to-use frameworks [8]. The contrary is the case, as feedback on behavioural skills remains challenging and requires trained faculty, even when applying robust frameworks [10, 28,29,30,31]. The process of becoming trained and experienced faculty in observing behavioural skills can be envisaged to span many years. However, such ‘expertise development’ studies regarding the process of behavioural skills feedback (i.e. observation, interpretation, evaluation and feedback) are scarce [32].

To summarise, experienced faculty is essential for the observation of behavioural skills and feedback and therefore team learning. However, sufficient knowledge of contemporary practice on the observation of behavioural skills by simulation facilitators, as well as potential areas for improvement is still limited.

Objectives

This study aims to answer the following overall research question:

• ‘What are the self-reported current practices and difficulties regarding observation of NTS amongst facilitators during team training and how have they been trained to observe these skills?’

The study addresses the following three sub-questions:

• (1) Which aspects of NTS find health care facilitators most important and/or most difficult to observe and why?

• (2) What strategies and tools (e.g. checklist) do health care facilitators utilise in their daily practice?

• (3) How are health care facilitators trained for observing NTS and what kind of future training needs do they have?

Methods

Study design

This study used a cross-sectional methodology using a multi-linguistic online survey within Europe. The survey and all corresponding documents (i.e. participants’ information) were made available in four languages (Dutch, English, German and French).

Participants

Facilitators supervising NTS team training sessions—previously mentioned SimZones 3 or 4—in healthcare were allowed to enrol in the study. Since the SimZones classification model might not be widely known, inclusion criteria were described as being a ‘teacher’, ‘instructor’ or ‘facilitator’, who provides feedback on non-technical skills during medical team training, using a scenario-based ‘immersive simulation’ in a (near) authentic setting.

Development and testing

The survey was constructed based on available literature regarding faculty development and observation of non-technical skills as well as interviews with five experts: an experienced psychologist in human factors and behavioural skills (WvL), two experienced simulation facilitators (AB, WvM) and two educational specialists (DV, DD). The survey was pilot-tested by seven participants who were excluded from participation in the final survey. Two double questions were split into separate questions, several small textual changes were made and one question was removed due to inconsistent interpretation.

The final survey consisted of five clusters: (1) demographic data, (2) level of experience, (3) observational aspects and difficulty, (4) observation strategy and (5) level of education and training. The survey consisted a total of 42 questions (including the consent questions), with the following distribution: fourteen multiple choice questions, nine open-text questions, twelve Likert-scale questions, three slider-scale questions and four matrix table questions. The exact number of questions presented to the participant was dependent on the provided answers as ‘adaptive questioning’ would only present relevant questions based on previous given input to minimise drop-out.

Translations were done by three different native speakers who all had academic language proficiency (i.e. Dutch-French, Dutch-German and Dutch-English).

Ethical review board and informed consent

The study was approved by the independent Dutch Association for Medical Education Ethical Review Committee (ID 2019.1.5).

The participant’s information letter was included in the distributive e-mails sent out to the participants. This letter included the aims and goals of the study, participants’ anticipated efforts (anticipated survey completion of 15 min), participants’ rights for withdrawal, data handling, ethical approval, research team information and contact details. At the start of the survey, participants were presented with an option to review this information letter, followed by an active opt-in informed consent question. Participation was anonymous; all demographic data used for descriptive analyses regarding age group, gender, background, country of employment and type of their organisation were voluntary questions.

Data collection

The survey was distributed as an ‘open survey’ via e-mail through local, regional and national simulation centres and organisations using a non-discriminative snowball sampling method. The survey link was open for responses between May and November 2019. Participants had 1 week to complete their survey. Afterwards, the system would close their record and mark their attempt as ‘incomplete’. Partially completed surveys were included for analysis to limit non-response bias. Raw data were stored in Qualtrics online platform (Qualtrics™, Provo, UT, USA). Respondents were prevented from taking the survey more than once using Qualtrics build-in functionality, which used cookies to prevent users from accessing the survey link more than once.

Data analysis

Quantitative data were analysed using descriptive statistical analysis using IBM Statistic Package for the Social Sciences (IBM Corp.© SPSS Statistics for Windows, Version 28.0.0.0). A data entry record with no questions answered past the informed consent and demographic section was regarded a dropout. Level of experience was calculated as the amount of days a facilitator had observed non-technical skills. Exploration of the association between level of experience and Likert-scale results were analysed using Spearman’s Rho test. Qualitative data on open-ended questions were analysed by applying open, axial and selective coding by two researchers (GB, LM) independently using an iterative, constant comparison approach. Any discrepancies in coding were discussed until a consensus was reached.

Results

After a 7-month period, 175 people had participated in the survey. Twenty-seven participants were excluded either because of dropout (N = 23, 13%) or non-observation of NTS (N = 4, 2%), leaving a total of 148 responses for analysis (Fig. 1).

Fig. 1

Overview of responses

Table 1 provides an overview of participants’ demographic data. The sample included a variety of facilitators, both men (51.3%) and women (48.0%) aged between 20 and 69 years. Medical doctors (48.6%) and (registered) nurses (41.2%) made up the vast majority of respondents. Participants were employed in over seventeen countries, with the majority working in the Netherlands (51.4%). Most respondents were employed (only) in university medical centres (44.6%), teaching hospitals (15.5%) and teaching facilities (23.6%) such as simulation centres or universities of applied sciences. The participants’ experience, calculated as (median, IQR) number of NTS observation days was 216 days [80–520].

Table 1 Characteristics of the participants

Self-reported importance and difficulty of NTS

Table 2 provides the results of the respondents’ perceived importance and difficulty of various NTS items. Perceived importance (scales 1–5; mean [SD]) ranged from very important for ‘coping with stress and fatigue’ (score 3,7 [0.8]) to extremely important for ‘communication’ (score 4,8 [0.4]). Perceived difficulty on the other hand (scale 1–5; mean [SD]) ranged from slightly difficult for ‘leadership’ (score 1,8 [0.8]) to moderately difficult for ‘coping with stress and fatigue’ (score 3,1 [1.1]). Spearman’s rho analysis showed no association between the level of experience and any of the NTS items.

Table 2 Perceived importance and difficulty of different NTS items

Participants were asked to provide an explanation on why certain non-technical skills were perceived—slightly to moderately—difficult to observe. Qualitative analysis of these responses (N = 93) revealed three major themes as shown in Table 3.

Table 3 Arguments for perceived difficulty in NTS observation

Theme 1: Not everything can be observed

Not all aspects of non-technical skills are expressed to the facilitator, and therefore, some cannot be observed directly. As an example, ‘mental processes’ or ‘mental frames’ of the participants are of utmost importance but cannot be observed directly. Also, the scripted training session itself might limit the expression of certain non-technical skills, e.g. due to time constraints or a particular focus of the training (i.e. scenario script).

Theme 2: Not everything is observed

Even if non-technical skills are expressed (either verbally or nonverbally), they can be missed easily by the facilitator, e.g. because the facilitator is focusing on a (pre)specified other aspect. It might also occur due to the subtlety of the expression, multiple expressions at the same time, hearing difficulties (e.g. audio difficulties of the simulation setting) and/or fatigue of the facilitator.

Theme 3: Interpretation of the observed NTS is difficult

Even when expressions or behaviours are observed by the facilitator, subsequent interpretation and analysis are difficult. Reported reasons for this are as follows: (1) interpretation is a ‘subjective process’, influenced by the facilitators’ own frame of reference and/or experience; (2) not all NTS have clear standards and/or observable criteria, e.g. ‘attitudes’ are reported to be (more) difficult to observe due to this phenomenon; (3) interpretation might be complex as performance fluctuates over time during the training session; (4) non-technical and technical skills might be intertwined hampering cause-effect interpretation and (5) participants might express different behaviour compared to their authentic workplace setting.

When participants were asked to indicate the percentage of observed NTS (in relation to the amount of NTS being expressed) in their average training session on a sliding scale (0–100%), and the mean [SD] result is 57% [17.6]. Furthermore, the mean [SD] number of team members facilitators observed, outnumbered their personal reported maximum 5.5 [2.0] and 3.9 [1.8], respectively; t(122) = 8323; p < 0.001). This finding was universal amongst all facilitators and not associated with the level of experience within the sample population (ρ = 0.084, n = 122, p = 0.360).

Facilitators were asked to elaborate on the barriers and enablers regarding the observation of NTS. Qualitative analysis (N = 106) revealed some overlap with the observational difficulties previously described. In addition, ‘other concomitant tasks’ such as role play in the scenario (i.e. patient, senior staff member) or observation of technical skills were most often cited as important barriers. Quote: [Translated] ‘You have to perform multiple tasks as a facilitator: provide information [to the participants], [scenario] role play, control the technology [i.e. mannikin] and make observations, so you cannot pay full attention to the observation of NTS.’ [R43] Reported enablers on the other hand included the use of multiple (co-)observers, having predetermined learning goals to focus on during the observation and making use of video recordings. Quote: [Translated] ‘There is a lot happening in a very short time span and I also need to focus on the content. I am always glad there are two of us.’ [R21] Furthermore, personal knowledge and experience were mentioned as well for identifying relevant behavioural skills: e.g. ‘It’s very hard to pick up on everything, training [and] practicing help[ed] to improve skills at recognition.’ [R115].

Observation strategies and tools used for observing NTS

Several strategies (e.g. co-observer, predefined learning goals) and tools (e.g. checklists/frameworks) can be utilised while observing NTS. The frequency by which health care facilitators used these is visualised in Fig. 2 (see also Supplementary Table S1).

Fig. 2

Strategies and tools used in the observation of NTS

A blank notepad, co-observers and predefined NTS learning goals were used most frequently. The use of a checklist/framework varied widely: 18% always, 29% most of the time, 14% about half the time, 20% sometimes and 19% never. All participants stating they used a checklist/framework at least sometimes were asked to specify this, including whether they had received formal training in its application (see Supplementary Table S2). The percentage of trained faculty was under 50% (range 0–45%) for each framework or checklist, with the exception of the 15-item CRM framework by Rall et al. [33] (12/14, 86%).

Participants were asked about the perceived usefulness of frameworks for their observation practice. Answers varied from: ‘not at all useful’ 5%, ‘slightly useful’ 23%, ‘moderately useful’ 26% and ‘very useful’ 31% to ‘extremely useful’ 15%. An additional open-ended question was asked to explore this further (see Table 4). Reported benefits of using a checklist/framework were the provision of a structure, a more objective observation and beneficial for research purposes. Reported disadvantages on the other hand were hindering of real-time observation, not suitable for every observation, less useful when more experienced, subjectivity of framework interpretation and resulting in a shift from learner-to-instructor-centred debrief.

Table 4 Pros and cons of framework use in NTS observation

Expertise development in NTS observation

Participants were asked to specify their attended training modalities with regard to their expertise development in NTS observation. Observing peers, self-study/own experience and direct supervision by expert(s) were selected most often by 64%, 62% and 45% of participants, respectively. The usefulness score (mean, [SD]) on a 5-point Likert scale was highest for direct supervision by expert(s), 4.21 [0.67], as shown in Table 5.

Table 5 Expertise development in NTS observation

Participants were given the option to comment on what aspects were most contributing to their learning curve with regard to the observation of NTS. Qualitative analysis (N = 53) is in line with the quantitative data as ‘reflection on one’s individual observation, either by peers or experts’ is mentioned by 49% (26/53). Quote: [Translated] ‘Observing the same scenario together and discussing what you have seen [is most useful].’ [R25].

Overall, given their faculty development, participants perceived themselves as ‘moderately competent’ for observing NTS (mean [SD] 3.42 [0.71]). There was no association between the level of experience and self-perceived overall competence (ρ = 0.122, N = 120, p = 0.185).

Finally, participants were asked whether they had future training needs (Question: ‘Are there any aspects you would like (additional) training for regarding the observation of NTS’). A total of 48% (57/119) answered positively whereas 52% (62/119) declined on having future training goals. There was neither an association between future training needs and self-perceived overall competence (r_pb − 0.125, N = 119, p = 0.099), nor with the level of experience (r_pb 0.128, N = 119, p = 0.167). Respondents who answered positively on having future training aspects were asked to specify these in an open-ended question. The majority of these answers related to either receiving direct feedback from other peers/experts, e.g. [Translated] ‘… More direct supervision and observation from trainers’ [R18] and [Translated] ‘I would like to see a training with side-by-side NTS analysis by participants and experts of some pre-recorded video fragments.’ [R25] or related to ‘training benefits’ in general, e.g. [Translated] ‘Extra training is always beneficial’ [R20]. Two other interesting quotes stood out: ‘I would like to have [had] formal training during my early practice as [an] observer.’ [R120] and [Translated] ‘…repetitive NTS observation training with some certification as many people are currently observing NTS without any proof of education/training’. [R36].

Discussion

This study endorses the opinion that observation of non-technical/behavioural skills is a complex task also within the medical community. The main results of the three explored research questions will be discussed consecutively in the sections below.

Medical facilitators regard all of the behavioural skills at least ‘very important’ to observe. Given the corresponding (only) slight to moderate difficulty with these observations, one can think that observation of behavioural skills is at first sight a simple task. However, when questioned more deeply, three themes were distilled, explaining why observation of behavioural skills is a rather challenging task. First, not everything can be observed; mostly mental frames of the participants cannot be observed directly and need to be explored based on the observed subsequent behaviour which is in line with publications recommending facilitators to question participants’ rationale for their actions in the debrief (i.e. ‘double-loop learning’) [18, 24, 34,35,36,37]. Second, not everything is observed; despite the expression of a behavioural skill, it might not have been observed by the facilitator. This might be intentionally by a distinct focus of the facilitator or unintentionally due to the subtlety or simultaneously of other behavioural skills or from distractions of concomitant tasks. Interestingly, facilitators report that the amount of team members they observe on their average training outnumbers their personal maximum. Salas et al. [27] advocated already that one facilitator should observe only two team members in complex teams so as not to overlook any interactions. With the average amount of team members being almost twice as high, it might not be a surprise that facilitators self-report they capture only half of the expressed behavioural skills [38].

Third, even when behavioural skills are observed by medical facilitators, interpretation is difficult, e.g. due to the personal frames of reference from the facilitators. This difficulty is in line with previous publications on limited inter-rater agreement, especially in larger, more complex, multidisciplinary teams [39, 40]. Gingerich et al. [41] however described nicely how to look at this facilitators’ cognition from different research perspectives, including the meaningful idiosyncratic stance, stressing that different interpretations between facilitators are not necessarily a bad thing and combining information from multiple observers and perhaps even participants itself can offer advantages for the debriefing phase [27, 41,42,43,44]. The technique of having multiple observers for ‘co-debriefing’ is nicely described by Cheng et al. [45] and appears to be used often by facilitators in our study.

The majority of facilitators had predefined behavioural skills learning goals at the start of their training. This allows for a tailored scenario selection and provides the facilitator with a clear aspect to focus on, also recommended by Salas et al. [27]. Although many checklists/frameworks are available for observation of behavioural skills, its use varied particularly within our population as did their reasoning for (not) using them. We hypothesised that novice facilitators would use some checklist/framework to structure their observation; however, we could not find an association between checklist/framework use and level of experience. We might have missed ‘true novice’ facilitators and/or the perceived disadvantages of checklist/framework use outweigh their perceived benefits in everyday practice. Interestingly, of all the participants that used a checklist/framework, only a minority used well-validated frameworks published in the literature, a phenomenon also addressed by Jenkins [46]. Frameworks can indeed have conflicting properties as reported in our study, i.e. sub-team specific frameworks (e.g. non-technical skills for surgeons [47], scrub practitioners [48] or anaesthetists [49]) have more clearly described behaviours, helping facilitators’ interpretation whereas ‘whole team’ frameworks (e.g. clinical teamwork scale [50], coordination behaviour in acute care teams [51]) are wider applicable possibly benefiting facilitators by gaining experience quicker. Jepsen et al. [8] provided a nice overview of 23 frameworks applicable to the observation of behavioural skills. Either way, facilitators need to provide their learners with detailed feedback to enhance meaningful learning [38] and behavioural frameworks might serve as a ‘cognitive aid’ and prevent facilitators’ fixation. On the other hand, if the observational task is already perceived as too challenging, having to complete yet another (concomitant) task might feel burdensome rather than helpful. Although the impact of checklists/frameworks on feedback and student learning is beyond our scope, using identical frameworks with multiple observers, it could facilitate an inter-rater discussion, thereby enhancing facilitators’ insight into their personal frame of reference.

When it comes to expertise development in behavioural skill observation, the majority of the facilitators were auto-didacts, who acquired their observation skills through either self-study/experience or from observing peers. Given the facilitators’ overall self-perceived moderate competence score coupled with the likely selection bias, it is interesting that only half of the facilitators reported future training desires. A study exploring simulation instructors’ self-reported needs, revealed that a ‘sense of competence’, i.e. achieving their highest level of professional functioning through self-reflection, was indeed present amongst many [52]. Since the vast majority of our sample population was made up of doctors and nurses being actively employed in health care facilities, time constraints and/or workload might be a relevant hypothesis to examine further, especially when striving for (regularly) trained faculty [53, 54]. Deployment of nonclinical faculty has been proposed to overcome time constraints for expertise development and re-certification within the medical domain [53, 55].

Strengths and limitations

To the authors’ knowledge, this study is the first to address self-perceived current practices and expertise development of behavioural skills observation amongst health care facilitators. The study provides relevant insights for different behavioural (research) domains such as expertise development, implementation of observational frameworks and organisation of (medical) team training. In order to focus on observation and interpretation of behavioural skills, we purposefully did not include questions on team feedback (i.e. ‘debriefing’). One of the important disadvantages of this study methodology is the inability to validate self-reported data. In addition, the cross-sectional sample strategy most likely skewed our sample population towards the more intrinsically motivated facilitators. Quantification of a response rate to analyse this further is unfortunately not possible since there is no formal registration for health care facilitators observing behavioural skills within Europe. In addition, half of the sample population was employed in the Netherlands, although we could not find this to be a confounding variable in the results.

Future research directions

More insight into what determines qualitative behavioural skill observation would allow tailoring of facilitators’ expertise development and guidelines for everyday practice. For example: in what way does the facilitator-team member ratio influence the quality of the observed behavioural skills? Is the amount of self-reported observations by the facilitators correct, and if so, what behavioural aspects are observed and why? In addition, medical facilitators tend to be mostly auto-didacts who appreciate co-observation with peers or experts. Research analysing how facilitators learn from each other and experts while observing medical teams can further benefit expertise development.

Future practice directions

Meanwhile, the three themes distilled in this study hold important implications for everyday practice. First, since not everything can be observed directly, facilitators need to reserve sufficient time for the (learner-centred) debriefing to unravel the—unobservable—participants’ thoughts. Second, the shortcomings of the human facilitator should be acknowledged. Facilitators cannot fulfil concomitant tasks in addition to observing behavioural skills and might even limit their attention further to (only) a (pre)specified aspect. This also necessitates multiple faculty members to run a (complex) team training, such as an ‘operator’ for controlling the mannikin and multiple co-observers. Despite this, facilitators need to be aware of their personal frame of reference compared to that of others.

Conclusions

Observation of behavioural skills by facilitators in health care remains a complex and challenging task. Facilitators’ limitations with respect to attention, focus and (in)ability to do concomitant tasks need to be acknowledged. Although strategies and tools can help to facilitate the observation process, they all have their limitations and are used in different ways.