Observer roles that optimise learning in healthcare simulation education: a systematic review

Nine studies were selected from the 5469 potential papers identified using the PRISMA process [13] (Fig. 1). The studies are summarised in Table 3. The included studies used quantitative, qualitative and mixed methods. A modified version of Buckley’s quality indicators, devised for assessment of quantitative, qualitative and mixed methods studies was selected as the quality assessment tool [14]. These 11 quality indicators relate to the appropriateness of study design, conduct, results analysis and conclusions and are not biased towards any particular research methodology (Table 4).

Reference	Research paradigm, design & samplinga	Participants	Intervention	Learner Observation Style	Results
Bell, Pascucci, Fancy, Coleman, Zurakowski and Meyer [24]	Mixed methods	Health professionals from four disciplines (n = 192)	Use of improvisational actors in difficult conversations to teach communication and relational skills to practicing health professional	Non-directed role: no use of observational tool or verbal guidance reported	No difference between observers and hands on learners in: perceived realism; usefulness of actors; usefulness of scenarios; and, opinions on non-actor role play
	Post-simulation survey design with qualitative and quantitative analysis	Teaching faculty (n = 33)
	Convenience sample	Actors (SP) (n = 10)
		Hands on participants (47 %)
		Observers (53 %)
Harder, Ross and Paul [25]	Ethnographic stud	Bachelor of Nursing students year 3 (n = 84)	Role assignment within regular simulation session with analysis of experience and perceptions of learning within different role	Non-directed role: no use of observer tool or verbal guidance reported	Students preferred assignment to nursing roles rather than observer or non nursing role
	Observational design with focused interview and journal review of selected participants	Participant/observation (n = 84) interview (n = 12)	All participants experienced both roles		Structured role descriptions positively affected learning outcomes
	Volunteer sample	journal review (n = 4)
Hober and Bonnel [11]	Qualitative	Bachelor of Nursing “senior” students (n = 50)	Immersive simulation scenarios with students randomly assigned to active or observer roles	Directed observer role: observer tool – educator provided activity guidelines	Observer role beneficial, less stressful
	Survey and interview design	Observers (n = 23)	All completed survey		Use of a guided observer tool useful
	Convenience sample	hands on learners (n = 27)	Observers interviewed		Able to reflect in action and on action
Kaplan, Abraham and Gary [27]	Quantitative	Bachelor of Nursing “junior” students (n = 92)	Immersive simulation scenarios -	Directed observer role: observer tool -checklist	No difference in knowledge
	Randomised groups	Observers (n = 46)	participants self selected roles
	Convenience sample	Scenario participants (n = 46)	Unclear whether observers self selected or were assigned		Limited as aggregated post satisfaction survey data
			Post scenario knowledge test and satisfaction survey
Lau, Stewart and Fielding [22]	Quasi experimental randomised to roles	Medical students (bilingual) year 1 (n = 160)	Student role plays with comparison of learning between interpreter role play and observer role	Directed observer role: observer tool -checklist	Observers rated post knowledge higher than learners in interpreter role-play
	Convenience sample		Self rated pre & post knowledge
Smith, Klaassen, Zimmerman and Cheng [26]	Mixed methods with increasing variables over three years	Bachelor of Nursing “junior” students	Introduction of simulation year 1	Non-directed role: no use of observational tool or verbal guidance reported	No significant difference in learning outcomes, student perceptions or peer evaluations
			Introduction non nursing participatory roles year
	Convenience sample	year 1 (n = 67)	Introduction non participatory observer roles year 3
		year 2 (n = 72)
		year 3 (n = 85)
		Note only the year 2 and 3 data were included in review
Stegmann, Pilz, Siebeck and Fischer [20]	Quantitative	Medical students (n = 200)	Comparison of participatory role and observer role in simulated patient scenario with and without observation tool	Non-directed and directed observer roles compared: checklists and feedback scripts used	Observational learning (especially if supported by observer script) more effective than learning by doing
	Crossover design 2x2x2 pre-test post-test
Stiefel, Bourquin, Layat, Vadot, Bonvin and Berney [23]	Quantitative	Medical students (masters level) (n = 124)	Individual training with simulated patient encounter	Non-directed role: no use of observer tool or verbal guidance reported	Measured outcomes no difference
	Randomised into 2 group	Individual training (n = 49) Group training (n = 75) -participated in simulation (n = 14) observed (n = 61)	Group training with simulated patient encounter		Those who observed but did not participate felt they did not meet their learning objectives as well compared to the other 2 groups
	Evaluation using instructor rating scale and student questionnaire		Group training with observation of simulated encounter
	Convenience sample
Thidemann and Soderhamn [21]	Quasi experimental	Bachelor of Nursing student year 2 (n = 144)	Immersive mannequin simulation with random allocation to groups	Directed observer role: observer tool with specific task focus	Post-test scores higher in all groups independent of rol
	Pre - and post-simulation knowledge test and student questionnaire		Four volunteers within each group allocated to participatory and in scenario observer roles – remainder observers (n = 72)		More satisfaction with nurse role
	Convenience sample over two consecutive years

^aas attributed by author where available

Criteria (Yes, No, Unclear)	Bell	Harder	Hober	Kaplan	Lau	Smith	Stegmann	Stiefel	Thidemann
Note: rater disagreement shown as two scores
Clear research question	U	Y	Y	Y	U	Y	Y	U	Y
Subject group appropriate for study	Y	Y	Y	Y	Y	Y	Y	Y	Y
Reliable and valid methods (qualitative or quantitative) used	Y	Y	Y	Y	Y/U	Y	Y	Y	Y
Completeness of data (drop out, questionnaire response rate >60 %, attrition rate <50 %)	Y	Y	Y	Y	N	N	Y	N	Y
Controlled for confounders or acknowledged if non RCT design	N	U/N	N	U/N	U	N	Y	U	Y
Statistical and other analysis methods appropriate	Y	Y	Y	Y	Y	Y	Y	Y	Y
Data justifies the conclusions drawn	Y	Y	Y	Y	N	Y	Y	U/N	Y
Study could be replicated	Y/U	Y	Y	Y	U	N/U	Y	Y	Y
Prospective study	Y	Y	Y	Y	Y	Y	Y	Y	Y
Relevant ethical issues addressed	U	Y	Y	N	U	Y	Y	U	Y
Triangulation of data	Y	Y	Y	Y	N	Y	Y	N	Y
Total Score/11 (lowest score reported)	7	10	10	9	3	8	11	5	11

Two reviewers (SO, EM) rated the quality of the studies with an inter-rater agreement of 0.94 across 99 data points. Seven studies meeting seven or more criteria as specified by Buckley, were considered high quality studies [14]. There was a wide range of quality with scores from 3 to 11 out of a possible 11. Most common problems encountered were with data completeness, control for confounders, study replicability and addressing ethical issues. Two studies, Stegmann [15] and Thidemann, [16] met all 11 criteria. Two studies, Lau [17] and Stiefel, [18] met six or less criteria. Rater differences are shown in the table as two scores, with the lowest total score reported where there was a discrepancy (Table 4).

Eight of the nine studies compared knowledge, skills, attitudes or behaviours between the hands-on role and the observer role [11, 15, 17, 18, 20–22]. Six studies used a pre and post-test design, three of which were self-assessment of improvement in knowledge and/or skills [17–19] and three studies tested knowledge [15, 16, 21]. Two studies examined knowledge in a post-test only design [22] one of which was a self-assessment [20]. Outcomes included knowledge (six), ‘non-technical skills’ (eight), technical skills (three), attitudes (two) and behaviours (one).

Four studies found no difference in outcomes between the hands-on learners and the observers [11, 16, 19, 22]. Two studies reported superior outcomes in the hands-on group [18, 20] and one study reported better outcomes in the observer group [15]. The study that found superior outcomes for the observer group and three of the four studies that found no difference in outcomes between the hands-on and observer groups [15–17, 22] incorporated an observer tool to guide the observer group. Neither study that demonstrated superior outcomes by the hands-on learners employed an observer tool [18, 20].

Six studies considered the perceived value of the hands-on learner and observer roles to the participants. Two studies reported that participants valued the hands-on roles more than the observer role [18, 20], one study highly valued the observer role [16] and three studies reported no difference in the value of the roles [11, 19, 22]. Two of the three studies with no value difference in roles [11, 22], and the study that valued the observer role highly [16] used an observer tool. The study that valued the hands-on roles higher did not employ an observer tool for the observer group [18]. The observer tools included performance checklists [15, 17, 22], feedback or observation guides [11, 15], or observer role instructional briefing [16]. All studies except Bell [23] documented including observers in the post simulation debrief or feedback.

We sought reported factors that contribute to the optimisation of learning in the observer role. It is clear from this review that the use of observer tools to focus the observer and role clarity are strongly associated with role satisfaction and learning outcomes in observer roles. This finding is supported by Bandura’s social learning theory and Kolb’s experiential learning cycle and we propose that these form the basis of the directed observer role.

One of the outstanding findings from this review is the association of observer tools with both satisfaction and equal if not better, learning outcomes in observer roles. The use of these tools may move observers from simply watching to actively observing. The activation of observers allows those in that role to experience the satisfaction and learning normally associated with hands-on experience. Simulation is described by Dieckmann et al as a social practice where people interact with each other in a goal orientated fashion [24]. The observer tool provides this necessary goal orientation for observer roles. Directed observers are focused on the learning objectives of the simulation.

This is explained by Bandura’s social learning theory, which proposes that virtually all learning acquired experientially could also be acquired on “a vicarious basis through observation of other people’s behaviour and its consequences for them” [25]. Through observation learners can build behaviours without trial and error, experience emotions by watching others and resolve fears through other’s experience. Bandura describes this as a process of attention, retention, reproduction and motivation [25]. Bethards reports on a program where “simulation experiences are designed around the observer role using the four component processes of Bandura’s observational learning construct” [26]. They postulate that this provides all their learners, regardless of role, the same opportunities to achieve the learning objectives [26].

Vicarious learning requires active listening, reflective thinking and situational engagement [27]. Nehls describes this in the context of narratives; lived experiences shared for the purpose of learning [27]. The addition of “active watching” to Nehls’ definition fits well in the simulation context. In a review of vicarious learning, Roberts concludes that vicarious learning occurs during story telling and discourse, and may require a teacher to help find meaning [28]. In the context of scenario-based simulation the story is the scenario or case; active listening and watching is engaged with the use of tools or tasks and the reflective facilitated discussion is the debriefing. It seems important that for optimal learning to occur, observers be engaged in all aspects including the debrief.

Experiential learning is viewed as fundamental to simulation and clinical practice [29, 30] and the theoretical foundations of simulation are commonly described in terms of Kolb’s experiential learning cycle [29]. Kolb proposes a cycle of concrete experiences which on reflection are distilled into abstract concepts that can then provide the basis for future actions and further testing [31]. Kolb stresses that this is an unending cycle and educators need to be aware that learners have a preference for, and may enter at different stages of the experiential learning cycle, but need to be moved through the entire process. A dangerous presumption for educators and learners alike is that concrete experience requires hands-on participation. Vicarious learning theory and Kolb’s experiential learning cycle form the theoretical basis for directed observation.

It seems that observers with the appropriate tools can benefit vicariously from the experience of the hands-on learners. Simulation is a facsimile of the clinical environment so the findings here may also translate to observation in similar clinical practice situations. This directed observer role is different to indirect workplace learning described by Le Clus, where the emphasis is on observers seeking learning to meet their personal needs [32]. However, the concept of observer learning as a social practice aligns with both [24, 32].

Stegmann reports better outcomes from observers preparing to provide feedback than those completing a checklist or in a hands-on role [15]. The impending ‘debrief’ where observers have an expectation that they will be asked to contribute their opinions about the encounter may sharpen the focus of their observations. Bandura describes this as an external motivator [25]. This ‘heightened state’ may mean observers are more likely to engage in standards of practice required for the simulation (for example, measures of good communication) and consider how the simulation participant’s performance measures up to this standard. Thidemann used reporting on standards of practice in her directed observer role guidelines [16].

The learners who did not value observer roles as highly as a hands-on role described observer roles as passive, or boring [20]. They were not fully engaged in the learning process. Emotional engagement in simulation is connected to the feeling of relevance of the scenario to the goals of the session [24]. Lack of goal direction may have prevented observer engagement. It is not clear whether there is an optimal level of activation for learning in observer roles or whether it differs between learners. Learners that valued observer roles described it as being less stressful and providing them the opportunity to see the big picture, examine details from a distance, and provide meaningful feedback to the team [11]. Stress decompression, a feature of debriefing frameworks, is necessary for reflection [30, 33].

The ability to reflect is important in the provision of feedback. An understanding of performance requirements and a judgement regarding the observed performance and its relationship to the standard is required before bridging strategies can be formulated [34]. In directed observer roles, information was provided in the form of the observer tool (e.g. checklist) defining the standards and/or objectives for the learners. The directed observers were able to use these tools to observe, reflect upon and formulate their peer feedback for the debrief.

In-scenario observers, that is non-clinical or other professional roles within the scenario, reported that lack of scripts or clear direction detracted from the act of observation because of anxiety regarding role performance requirements [20]. These aspects of role fidelity have been identified as a barrier to student satisfaction with role play [35]. The other studies that used non-clinical or other non-congruent professional roles viewed these learners as hands-on participants and did not include specific findings for these in-scenario observer roles [17, 20, 21]. Thidemann commented that the nursing roles in their scenarios were the most preferred roles [16]. The lack of clarity in the separation between professionally congruent and incongruent hand-on roles in these studies prevents drawing any real conclusions from the data. In a report of a large study for the National League of Nurses Jefferies and Rizzola ¹ concluded that whilst knowledge and self-confidence were unrelated to role allocation, there was a perceived lack of collaboration in the observer role and there was a responsibility for educators to provide structure for this to occur [9].

While learners have assessed the value of observer roles, there has not been a published assessment of the value placed upon observer roles in simulation by educators or facilitators. Use of observer tools or activities and the active involvement of observers in the post-scenario debrief could be considered an indirect indication of the value educators place on learning in observer roles.

It is also unclear as to whether there is a group of learners better suited to learning through observation than learning through hands-on participation in the simulation. Whilst most of the studies used role allocation, one study [20] had a portion of study participants who either self allocated or worked through the case as a group without assigned roles. There was confusion amongst the students in this study as to which roles were considered to be observers; for example some students viewed the documentation nurse as an observer role while others viewed it as a hands-on role. No studies examined whether self-allocation to roles would result in better learning outcomes. The reasons behind self-allocation were also not examined and may be worthy of further study.

An important area for further study includes establishing educator perceived value of observational roles, and the potential impact of these perceptions on simulation education design and orientation of learners to roles within the scenarios. Activation and emotional engagement in the observer role has also not been explored, and provides future research potential.

Learning outcomes for participants and observers in simulation can have value if all roles involve active learning either through hands-on roles within the simulation, or through use of tools to facilitate active observer learning. The value that students attach to observer roles seems to be related to the value educators place on them as evidenced through role briefing, use of observer tools to hone judgement of performance compared to standards, and inclusion of observers’ perspectives in debriefing.