The authors registered the study at Clinical.Trial.Gov (ID n. NCT04395963).
Forty PGY4-5 anesthesia trainees volunteered from the University of Catania Medical School to be enrolled in this prospective, observational study. Each participant gave informed written consent, and privacy, confidentiality, and anonymity were fully guaranteed by the EESOA Research Board.
In our region, simulation centers do not have access to a formal ethical approval process.
Our simulation center adheres and follows the Healthcare Simulationist Code of Ethics supported by the Society for Simulation in Healthcare [9]. Our study was eligible for exemption, in accordance with US Federal Human Subject Regulations–Protection of Human Subjects, due to the nature of the study itself, as no patients were involved, the trainees participating were volunteers, the researchers ensured that those taking part in the research would not be caused distress. All participants’ personal and other data were completely anonymized, and all the investigators had no conflict of interest and were not involved in any of the participants’ university teaching programs.
We studied eight teams, each containing five participants, and the scenario was repeated four times in order to note any difference in behavioral performance in the participants who were given the role of leader. Our study was a typical “high-fidelity simulation with role exchange” [10]. It is well-known how exchanging professional roles helps professionals understand and “put themselves in the shoes” of their colleagues. This technique has a high didactic value as it trains trainees to better understand the points of view of other healthcare professionals participating in the emergency. For the purpose of the study, we randomly assigned the “leader’s role” to the same subject and rotated the others during the four scenarios (assigning in turn the roles of midwife, obstetrician, nurse, and anesthesia trainee) in such a way that at the end of the rotation, each of them had participated with a different role.
For the leader’s role, we selected those who had the most experience in obstetric anesthesia, based on the time spent in the delivery room during their curricular rotations. Thereafter, we randomly gave them the role of “senior anesthesiologist” in each scenario, making sure that among the different roles assigned, the one of “senior” was the only one they interpreted. In this way, we expected that the participant who was assigned the role of “senior anesthesiologist” (and who in fact had the most experience in obstetric anesthesia) would take the leadership. In the case of shared leadership with someone else, the case was not included in the study.
Every trainee who was assigned the leader’s role wore the eye tracking glasses during the scenario.
For this study, we used a commercially available Tobii Pro Glasses 50 Hz wearable wireless eye tracker. This system can measure eye movements using cameras integrated into the eyeglasses which record the corneal reflection of infrared lighting to track pupil position, mapping the subject’s focus of attention on video recordings of the subject’s field of vision (gaze). In addition to tracking gaze, this system also enables the measurement of various eye metrics including fixation frequency and dwell time, used as a surrogate measure of perceived stimulus importance [11].
All the eye-tracked procedures were recorded immediately after accurate individual calibration, during which the participant, after wearing the glasses unit, focused on the center of the calibration target.
All the eye tracking video recordings were stored and analyzed using the Tobii Pro Lab Software. We selected 27 areas of interest (AOI) (Figs. 1 and 2), to define regions of a displayed stimulus and to extract metrics specifically for those regions as follows:
Eleven AOI concerning the simulation room: airway suction, anesthesia trolley, blood loss, drip phleboclysis, ECG monitor, clock, oxygen source, control room, phone, blood loss collector bag, and other (the space in general) (Figs. 1 and 2)
Eight AOI concerning the participants: anesthesia trainee, anesthesia trainee’s eyes, obstetrician,
obstetrician’s eyes, nurse, nurse’s eyes, midwife, midwife’s eyes (Fig. 2)
Eight AOI concerning the manikin: right arm (on which the sphygmomanometer was placed), the left arm (on which two intravenous accesses were set), right leg, left leg, belly, trunk, vagina, face (Fig. 3).
The number and duration of fixations for each area of interest were examined.
The fixation points were generated by the Tobii software’s filter according with the following parameters: max gap length 75 ms; noise reduction: window size (samples) 3; velocity calculator: window length 20 ms; merge adjacent fixations: max time between fixations 75 ms; max angle between fixations 0.5°; minimum fixation duration 60 ms.
Each fixation point was assigned manually to a specific AOI by an independent, blinded investigator (simulation technician specifically trained in eye tracking) who reviewed the video recording of each scenario.
The eye tracking metrics were mapped as gaze plots and heat maps. Heat maps and gaze plots are data visualizations that can communicate important aspects of visual behavior clearly and with great power. Gaze plots show the location, order, and time spent looking at locations on the stimulus. Time spent looking, most commonly expressed as fixation duration, is shown by the diameter of the fixation circles. The longer the look, the larger the circle.
Heat maps show how looking is distributed over the stimulus and can effectively reveal the focus of visual attention.
The evaluation of the behavioral skills of the leader and of the technical skills of the team was made by two expert observers not involved in the scenarios, who independently reviewed the video recordings of the scenarios.
The technical skills of the team were evaluated on the basis of the completion of a PPH checklist. For the design of this checklist, we reviewed PPH guidelines from recognized obstetric bodies and literature, relevant papers from the literature, and their institutional PPH protocol [12,13,14,15].
We then chose, by consensus, the final action items for the checklist, identifying 25 standardized key tasks for inclusion on the PPH checklist. We assigned one point for each task executed, for a maximum of 25 points. This checklist worked as the reference guide for pre-scenario briefing and for the team’s technical skills evaluation during the scenario (Appendix 1).
We also developed a standardized questionnaire for the evaluation of the behavioral skills of the leader (Appendix 2), derived from the Anaesthetists’ Non-Technical Skills (ANTS) behavioral marker system [16] and the Ottawa Global Rating Scale (GRS) [17]. Each independent observer assigned a score for leadership, communication, and situational awareness (Appendix 2).
Interobserver reliability was also calculated
No formal training took place before the first scenario, in order to consider the first scenario as the participants’ baseline performance. All the teams underwent standardized educational training on PPH Guidelines immediately before the second, third, and fourth scenarios.
The scenario consisted of a severe PPH (> 1500 mL blood loss) due to refractory uterine atony in a multiparous 28-year-old patient who had undergone a spontaneous vaginal delivery. The patient became tachycardic and hypotensive consistent with hemorrhagic shock. All simulations were performed in the simulation room of the EESOA Simulation Center (Rome) using a high-fidelity manikin (Sim Mom Maternal and Neonatal Birthing Simulator, Laerdal, Norway). All scenarios were videotaped. The scenario was stopped when each team had completed all 25 tasks of the checklist, or when 15 min had elapsed. Each scenario was followed by a standardized debriefing led by an expert debriefer.
A study investigator, expert in both PPH and simulation debriefing and not involved in the simulation activity, observed each scenario in the control room, to record and check the team’s performance (PPH evaluation and treatment), according to the established 25 PPH key tasks (Appendix 1). The leaders’ behavioral scores (Appendix 2) were assigned by two observers, experts in communication and evaluation in simulation and not involved in the scenarios, who reviewed the videos of each simulation.
After completion of the study, all the leaders were divided into two groups, depending on the scores received for their leadership behavioral skills during the scenarios, and their eye tracking metrics were compared. We divided all the leadership situations into two groups: the High-Performance Leader group, HPL, which included all the leaders who had received the highest scores (score = 5) and the Low-Performance Leader group, LPL, which included the leaders who had received the lowest scores (score: 1–2) during the four scenarios.
Statistical analysis
Data are presented as means, confidence intervals (95% CI), and standard deviations (SD).
The leaders’ performances were calculated by using the means of the scores given by each independent observer on leadership, communication, and situational awareness (Appendix 2).
In order to better discriminate the best and worst performance assessment, a linear transformation to convert the scale into a 5-point scale was used.
The eye tracking metrics were compared by using a two-way unpaired t test with lower and higher alternative hypothesis to compare the two groups.
The overall team performance (assessed by the PPH checklist) from the first to the fourth scenarios was examined by using the ANOVA test and Dunnett’s post hoc test.
It was not possible to calculate the sample size a priori because at the start of the study, it was obviously unreasonable to determine how many leaders would perform well or poorly.
The post hoc power analysis, set at a significance level of 0.95 and a calculated effect size for almost all comparison above 1, was in the range of 60–80% power.
The Cohen’s Kappa coefficient was applied to measure the degree of agreement between the two assessors (inter-rater reliability).
