Cumulative evaluation data: pediatric airway management simulation courses for pediatric residents

Objectives To utilize cumulative evaluation data of the pediatric airway management simulation-based learning course on knowledge and practical skills of residents in the Saudi Commission for Health Speciality (SCFHS) in order to measure its efficacy and areas for improvement. Methods The evaluation is a retrospective cohort study that compares pre- and post-test (knowledge and skills) of a pediatric airway management simulation course. The 2-day course has been conducted four times annually at CRESENT and is comprised of interactive lectures on airway management and crew resource management, a demonstration of fundamentals of intubation, three skill stations, and six case scenarios with debriefing. Our evaluation data includes all pediatric residents who attended the course between January and December 2015. Results Forty-six residents participated, of whom 30 (65.2%) are male and 16 (34.78%) are female. Overall, there is statistically significant improvement between the pre-test and post-test knowledge and practical skill scores. The pre-test scores are significantly different between the four different resident levels with p values of 0.003 and <0.001 respectively. However, there are no statistically significant differences in the post-test scores among the four different resident levels with p values of 0.372 and 0.133 respectively. The practical skill assessment covers four main domains. Improvements were noted in pharmacology (811%), equipment setup (250%), intubation steps (200%), and patient positioning (130%). The post-test scores are similar in all practical skill categories for the four different residency levels. Discussion Our outcome-based evaluation strategy demonstrated that residents met the course learning objectives. The pediatric airway management simulation course at CRESENT is effective in improving the knowledge and practical skills of pediatric residents. Although the greatest improvement is noted among junior residents, learners from different residency levels have comparable knowledge and practical skills at the end of the course. Things that can be improved based on our study results include stressing more the type and dosages of the medications used in airway management and mandating the course for all junior pediatric residents. Although residents scored well, specific knowledge and skill elements still led us to targeted areas for course excellence. Similar courses need to be integrated in the pediatric residency curriculum. Further research is needed to study skill retention and more importantly its impact on patients’ care. Although resource-intensive, the use of cumulative evaluation data helped to focus quality improvement in our courses.


Introduction
Airway management is a common procedure performed in the Pediatric Intensive Care Unit (PICU) and Emergency Department (ED). In contrast to the clinical experience with elective intubation in the operating room, intubation of critically ill patients has been associated with several complications [1]. Most airway management situations in the PICU/ED are emergent, leaving providers with limited time to perform a systematic airway assessment. Critically ill patients frequently have significant cardiac and pulmonary disease and limited physiologic reserve [2,3]. These complicating factors commonly result in significant preoxygenation difficulty, limitations in the choice and dose of commonly used induction and paralytic agents, and less time for intubation preparation and performance. Loss of muscle tone, secretions and upper airway edema also increase the technical difficulty of glottis visualization and successful procedure performance [4,5]. The number of intubation attempts increases the risk of adverse tracheal intubation associated events such as severe hypoxia, hypotension and cardiac arrest [2,6,7]. It also increases the risk of intraventricular hemorrhage in low birth weight neonates [8].
The pediatric airway management simulation course is conducted four times per year at CRESENT, King Fahad Medical City (KFMC), Riyadh, Saudi Arabia. The course was adapted from the American College of Chest Physicians (ACCP). The course has been selected by pediatric residents at KFMC in the top five most common simulation courses needed. We wanted to utilize cumulative evaluation data of the pediatric airway management simulation-based learning course on knowledge and practical skills of residents in the Saudi Commission for Health Speciality (SCFHS) in order to measure its efficacy and areas for improvement.

Methods
We used an outcomes-based approach of evaluation to inform future courses [9][10][11]. We chose to use evaluation data over a one-year period since our participant numbers are quite small and we wanted to ensure weight of data and accommodate several iterations of the course. Although the course is standardized there may be variations based on participant engagement. This evaluation design is a retrospective cohort pre-test post-test that compares knowledge and practical assessments of residents attending the pediatric airway management simulation course.

Course description
The two-day course is conducted four times annually at CRESENT. Twelve to 18 participants per course learn with an instructor to resident ratio of 1:6. Learning objectives are listed in Table 1. During the introduction, the course director introduces the instructors and simulation technicians. The course director and instructors are all pediatric intensivists with experience in simulation-based education including targeted training on using simulation to support learning. The faculty has participated in faculty development courses at CRESENT, namely the FD-Sim course, and IMS course from the Center of Medical Simulation (CMS). The residents tour the simulation center and are familiarized with the simulation rooms, debriefing rooms, simulators and all the equipment. The course director introduces the basic assumption and safety container [12]. The simulation rooms resemble PICU rooms and equipped with SimJunior® or SimBaby®, crash cart with a defibrillator and airway tools for infants and children. The course utilizes three skills station rooms: 1) basic airway tools and infant and child intubation heads (Fig. 1), 2) advanced airway tools such as video laryngoscope and intubation bronchoscope and 3) surgical airway tools for cricothyroidotomy utilizing TraumaMan® manikin.
The course schedule includes: 1) two interactive lectures on airway management and crew resource management, 30 min each, 2) a demonstration session on fundamentals of intubation for 60 min, 3) three skills stations, 4) six case scenarios, two on each concept of can ventilate-can intubate, can ventilatecan't intubate and can't ventilate-can't intubate. Each scenario is followed by video debriefing. All activities are done in group fashion. When the scenario necessitates, moulage is performed on SimJunior® or SimBaby®. Debriefing usually follows the 10-min case scenario and lasts for 20 min. Three phases of debriefing are used: 1) reaction discussion of both feelings and facts, 2) understanding on how to improve or sustain performance through exploring, discussion and teaching and generalization, and 3) summary and take home messages.

Evaluation instruments
Evaluation of participants is done at the beginning of the course and then at the end. Residents complete a pretest of 20 multiple choices questions with equal weight and a maximum score of 100%. The test was developed by one the authors (SA) to reflect the learning objectives and was face validated with a group of experts in the field. In the practical skills test, the resident is given a scenario of a patient with respiratory failure and to manage. A 31-point checklist with equal weight adapted from ACCP (Table 2) is used to assess the resident's performance. The checklist is divided into 4 parts: equipment setup, patient positioning, pharmacology and intubation steps.

Study population
The study includes all pediatric residents under Saudi Commission for Health Specialties (SCFHS) training program who attended the pediatric airway management simulation course between January 2015 and December 2015 at CRESENT, KFMC.

Statistical analysis
Categorical variables of gender and level are presented as numbers and percentages. Continuous variables of age, pre-test and post-test scores are expressed as Mean ± S.D. Paired sample t-test / ANOVA is applied to determine the mean significant difference among pre-test   Table 3. Overall, there is statistically significant improvement between the pretest and post-test knowledge and practical skills scores (Table 4). Eighty-seven per cent of residents show improvement in their knowledge test score, 13% had no change and none had a decline in their mean scores that is statistically significance (p<0.001). However, all residents, who participated in these courses, showed improvement in their practical skills. A comparative analysis of pre-test and post-test mean scores of the four categories of practical skills assessment showed significant statistical improvement with P-value <0.001 as presented in Table 5.
The breakdown of the pre-test and post-test knowledge and practical skills scores by resident level is presented in Table 6. The pre-test knowledge and practical skills scores are significantly different between the resident levels with p-values of 0.003 and <0.001. On the contrary, there are no statistically significant differences in the post-test knowledge and practical skills among the resident levels with p-values of 0.372 and 0.133.
The practical skills assessment is divided into four domains. The best improvement between pre-test and post-test is in the pharmacology 9% vs. 82% respectively (811%), followed by equipment setup 22% vs. 77% (250%), intubation steps 27% vs. 81% (200%) and finally patient positioning 20% vs. 46% (130%). The breakdown of practical skills domains by resident level is presented in Table 7. Only 2 categories of practical skills show statistically significant differences in the pre-test among the four residents' levels: equipment setup (p<0.001) and intubation steps (p<0.001). The post-test scores are similar in all practical skills categories for the different residency levels; equipment setup (p=0.168), patient positioning (p=0.815), pharmacology (p=0.093) and intubation steps (p=0.369).

Discussion
This study shows the importance and effectiveness of the pediatric airway management simulation course for pediatric residents under SCFHS training programs. The striking results are the improvement in all assessed categories of practical skills ranging from 130% to 800%, which makes a strong argument to mandate such courses to all pediatric residents. Several studies have shown the efficacy of airway management training on improving intubation skills [13,14]. However, the evidence of its impact on reducing the hazards and the risk on the patients remains limited [15].
A key element in assessing the effectiveness of simulation-based educational activity, is to document measurable improvement in knowledge, behavior and skills [16,17]. Unlike other courses, we have not relied on resident satisfaction with the course for quality improvement, but measured their knowledge and skills before and after the course. The detailed and comprehensive outcomes-based evaluation in this course provides sufficient data for us to maintain elements of the course and improve others. Things that can be improved based on our study results include: stressing  Table 4 is comparing pre-test and post-test pediatric airway management knowledge scores which showed a significant p value of <0.001, also it is comparing pre-test and post-test pediatric airway management practical skills scores which showed as well significant p value of <0.001   more the type and dosages of the medications used in airway management and mandating the course for all junior pediatric residents. On the other hand, by integrating the evaluation into the course schedule, it facilitates ease of data collection. It also has an orienting impact for all residents as the opening activity in the course. The practical skills assessment in particular is labor intensive; however, it is an imperative tool for accurate measurements of the course's impact. The course focuses on skills such as teamwork, crew resource management and communication techniques. These skills together with proper preparation of the intubation equipment, having them organized in predetermined way and the use of cognitive aid have crucial effects on the success of safe intubation [13,18,19]. Similar results have been reported with training of otolaryngology residents on advanced airway skills [20].
The pediatric airway management course at CRESENT targets all pediatric residents. There are clear differences in the pre-test scores among the four levels of residents which give validity to the assessment tool used. However, the junior residents show the greatest improvement in their post-test scores to the level of the seniors which strengthens the effectiveness of the course. Training for airway management including endotracheal intubation should be conducted early during residency to get the maximum benefit [21]. The importance of integrating airway management course into training programs for residents who manage critically ill children is essential as it reflects directly on patients' outcome and safety [15,17]. Attending advance life support courses once every two years is insufficient to improve intubation management [22]. A simulation-based education curriculum for a residency program is best constructed in a modular fashion [23]. A pediatric airway management course is one of these modules that best be administered early in the residency program.

Conclusion
The pediatric airway management simulation course at CRESENT is effective in improving the knowledge and practical skills of pediatric residents. Although the greatest improvement is noted among junior residents, learners from different residency levels have comparable knowledge and practical skills at the end of the course. Similar courses need to be integrated in the pediatric residency curriculum preferably at early stage of residency programs. Further research is needed to study skills' retention and more importantly its impact on patients' care. Our outcomes-based evaluation strategy has provided targeted insight to the strengths and areas for development in the course which we have acted upon.  Table 6 is comparing pre-test and post-test knowledge practical skill scores for each residency level separately which showed improvement of the scores at all residency levels, and by comparing the pre-test knowledge and practical skill scores, it showed significant difference between the four different resident levels with p values of 0.003 and <0.001 respectively. On the contrary, there were no significant differences in the post-test knowledge and practical skills among the four different resident levels  Table 7 displays the four main domains for the pre-test and post-test practical skill scores for each residency level separately which shows significant p values for all residency levels at all the domains, and by comparing pre-test scores among the four resident levels, only equipment setup and intubation steps showed significant p value of <0.001, while post-test scores showed almost similar scores for all the residency levels