Cumulative evaluation data: pediatric airway management simulation courses for pediatric residents
© The Author(s) 2017
Received: 27 December 2016
Accepted: 4 July 2017
Published: 1 August 2017
To utilize cumulative evaluation data of the pediatric airway management simulation-based learning course on knowledge and practical skills of residents in the Saudi Commission for Health Speciality (SCFHS) in order to measure its efficacy and areas for improvement.
The evaluation is a retrospective cohort study that compares pre- and post-test (knowledge and skills) of a pediatric airway management simulation course. The 2-day course has been conducted four times annually at CRESENT and is comprised of interactive lectures on airway management and crew resource management, a demonstration of fundamentals of intubation, three skill stations, and six case scenarios with debriefing. Our evaluation data includes all pediatric residents who attended the course between January and December 2015.
Forty-six residents participated, of whom 30 (65.2%) are male and 16 (34.78%) are female. Overall, there is statistically significant improvement between the pre-test and post-test knowledge and practical skill scores. The pre-test scores are significantly different between the four different resident levels with p values of 0.003 and <0.001 respectively. However, there are no statistically significant differences in the post-test scores among the four different resident levels with p values of 0.372 and 0.133 respectively. The practical skill assessment covers four main domains. Improvements were noted in pharmacology (811%), equipment setup (250%), intubation steps (200%), and patient positioning (130%). The post-test scores are similar in all practical skill categories for the four different residency levels.
Our outcome-based evaluation strategy demonstrated that residents met the course learning objectives. The pediatric airway management simulation course at CRESENT is effective in improving the knowledge and practical skills of pediatric residents. Although the greatest improvement is noted among junior residents, learners from different residency levels have comparable knowledge and practical skills at the end of the course. Things that can be improved based on our study results include stressing more the type and dosages of the medications used in airway management and mandating the course for all junior pediatric residents. Although residents scored well, specific knowledge and skill elements still led us to targeted areas for course excellence. Similar courses need to be integrated in the pediatric residency curriculum. Further research is needed to study skill retention and more importantly its impact on patients’ care. Although resource-intensive, the use of cumulative evaluation data helped to focus quality improvement in our courses.
Airway management is a common procedure performed in the Pediatric Intensive Care Unit (PICU) and Emergency Department (ED). In contrast to the clinical experience with elective intubation in the operating room, intubation of critically ill patients has been associated with several complications . Most airway management situations in the PICU/ED are emergent, leaving providers with limited time to perform a systematic airway assessment. Critically ill patients frequently have significant cardiac and pulmonary disease and limited physiologic reserve [2, 3]. These complicating factors commonly result in significant pre-oxygenation difficulty, limitations in the choice and dose of commonly used induction and paralytic agents, and less time for intubation preparation and performance. Loss of muscle tone, secretions and upper airway edema also increase the technical difficulty of glottis visualization and successful procedure performance [4, 5]. The number of intubation attempts increases the risk of adverse tracheal intubation associated events such as severe hypoxia, hypotension and cardiac arrest [2, 6, 7]. It also increases the risk of intraventricular hemorrhage in low birth weight neonates .
The pediatric airway management simulation course is conducted four times per year at CRESENT, King Fahad Medical City (KFMC), Riyadh, Saudi Arabia. The course was adapted from the American College of Chest Physicians (ACCP). The course has been selected by pediatric residents at KFMC in the top five most common simulation courses needed. We wanted to utilize cumulative evaluation data of the pediatric airway management simulation-based learning course on knowledge and practical skills of residents in the Saudi Commission for Health Speciality (SCFHS) in order to measure its efficacy and areas for improvement.
We used an outcomes-based approach of evaluation to inform future courses [9–11]. We chose to use evaluation data over a one-year period since our participant numbers are quite small and we wanted to ensure weight of data and accommodate several iterations of the course. Although the course is standardized there may be variations based on participant engagement. This evaluation design is a retrospective cohort pre-test post-test that compares knowledge and practical assessments of residents attending the pediatric airway management simulation course.
Learning objectives of the pediatric airway management course
1. Demonstrate observation of universal precautions at all times
2. Demonstrate clinical skills of competent performance of airway management
3. Define respiratory failure
4. Describe the basic anatomy and physiology of the paediatric airway
Practice-based learning and improvement
5. Demonstrate management of simple and difficult airway diseases
6. Demonstrate sound decision-making based on available medical information
Interpersonal and communication skills
7. Demonstrate the use of crew resource management
8. Demonstrate effective interdisciplinary teamwork
The course schedule includes: 1) two interactive lectures on airway management and crew resource management, 30 min each, 2) a demonstration session on fundamentals of intubation for 60 min, 3) three skills stations, 4) six case scenarios, two on each concept of can ventilate-can intubate, can ventilate-can’t intubate and can’t ventilate-can’t intubate. Each scenario is followed by video debriefing. All activities are done in group fashion. When the scenario necessitates, moulage is performed on SimJunior® or SimBaby®.
Debriefing usually follows the 10-min case scenario and lasts for 20 min. Three phases of debriefing are used: 1) reaction discussion of both feelings and facts, 2) understanding on how to improve or sustain performance through exploring, discussion and teaching and generalization, and 3) summary and take home messages.
Pediatric airway management procedural skills checklist
BVM with O2 on at 10L (positioned on left)
10cm PEEP valve (positioned on left)
Oral and/or nasal airway (positioned on left)
Free flowing IV
Suction on: coming from right, positioned on right
ET tube-proper size for age with stylet (positioned on right)
Cuff checked: 10cc syringe attached
ETCO2 detector (positioned on left)
Laryngoscope handle and blade with light on: positioned on left
Difficult airway cart immediately available
Side rails down
Head positioned to align airway axis
Bed height appropriate
Midazolam 0.05-0.1 mg/kg bolus or Fentanyl 1-2 μg/kg bolus ready or ketamine 1-2mg/kg AND Succinylcholine 1-2 mg/kg bolus or Rocuronium 0.6-1.2mg/kg bolus
Atropine or Epinephrine available
Put on personal protective equipment (gloves, mask, eye protection minimum)
Verbalize airway assessment
Intubator verbalizes explicit review of plan/back up with cutoffs
Induction agent given
Ability to ventilate considered/confirmed
Cricoid pressure applied (optional)
Neuromuscular agent given
Intubation technique appropriate
Intubator halts intubation efforts and initiates BVM if saturation decreases by 5% or <90%
Airway adjunct (oral/nasal) considered/employed if sat <90%
ETT set at appropriate length for age at gum line (3x tube size in mm)
Placement confirmed with 2 indicators (positive ETCO2, breath sounds, O2 saturation)
Intubator does not let go of tube until it is secured
The study includes all pediatric residents under Saudi Commission for Health Specialties (SCFHS) training program who attended the pediatric airway management simulation course between January 2015 and December 2015 at CRESENT, KFMC.
Categorical variables of gender and level are presented as numbers and percentages. Continuous variables of age, pre-test and post-test scores are expressed as Mean ± S.D. Paired sample t-test / ANOVA is applied to determine the mean significant difference among pre-test and post-test scores. A p-value of less than 0.05 is considered as statistically significant. All data is entered and analyzed through statistical package SPSS version 22.
The study is approved by KFMC IRB Committee.
Residency level of the trainees
Pre-test and post-test scores for knowledge and practical skills
48.0 ± 22.1
70.4 ± 15.5
17.5 ± 10.9
89.4 ± 9.6
Comparative analysis of pre-test and post-test mean scores of the four domains of practical skills
Mean ± S.D.
Equipment setup (10 points)
2.15 ± 1.66
7.74 ± 1.51
Patient positioning (4 points)
0.78 ± 0.96
3.54 ± 0.81
Pharmacology (2 points)
0.15 ± 0.36
1.63 ± 0.61
Intubation steps (15 points)
4.02 ± 2.52
12.28 ± 1.87
Breakdown of knowledge and practical skill scores by resident level
Knowledge test (20 points)
Practical skills (31 points)
R1 (n = 17)
6.8 ± 2.9
16.6 ± 2.7
4.3 ± 2.7
23.9 ± 3.6
R2 (n = 15)
10.6 ± 5.8
17.7 ± 2.3
7.2 ± 2.7
25.3 ± 3.6
R3 (n = 9)
11.4 ± 4.8
18.2 ± 2.1
8.9 ± 3.9
25.8 ± 4.4
R4 (n = 5)
15.2 ± 3.1
17.6 ± 2.3
13.2 ± 7.7
28.2 ± 1.7
Breakdown of practical skill scores domains by resident level
R1 (n = 17)
1.3 ± 1.2
7.3 ± 1.5
0.5 ± 0.9
3.5 ± 0.7
0.1 ± 0.3
1.4 ± 0.8
2.4 ± 1.5
11.8 ± 1.6
R2 (n = 15)
2.3 ± 0.9
7.7 ± 1.4
0.7 ± 0.8
3.5 ± 1.1
0.1 ± 0.3
1.7 ± 0.5
4.1 ± 1.9
12.4 ± 1.9
R3 (n = 9)
2.1 ± 1.7
7.9 ± 1.7
1.0 ± 1.3
3.7 ± 0.5
0.3 ± 0.5
1.8 ± 0.4
5.4 ± 1.9
12.4 ± 2.5
R4 (n = 5)
4.6 ± 2.5
9.0 ± 1.2
1.4 ± 0.9
3.8 ± 0.5
0.4 ± 0.6
1.7 ± 0.1
6.8 ± 4.1
13.4 ± 0.9
This study shows the importance and effectiveness of the pediatric airway management simulation course for pediatric residents under SCFHS training programs. The striking results are the improvement in all assessed categories of practical skills ranging from 130% to 800%, which makes a strong argument to mandate such courses to all pediatric residents. Several studies have shown the efficacy of airway management training on improving intubation skills [13, 14]. However, the evidence of its impact on reducing the hazards and the risk on the patients remains limited .
A key element in assessing the effectiveness of simulation-based educational activity, is to document measurable improvement in knowledge, behavior and skills [16, 17]. Unlike other courses, we have not relied on resident satisfaction with the course for quality improvement, but measured their knowledge and skills before and after the course. The detailed and comprehensive outcomes-based evaluation in this course provides sufficient data for us to maintain elements of the course and improve others. Things that can be improved based on our study results include: stressing more the type and dosages of the medications used in airway management and mandating the course for all junior pediatric residents. On the other hand, by integrating the evaluation into the course schedule, it facilitates ease of data collection. It also has an orienting impact for all residents as the opening activity in the course. The practical skills assessment in particular is labor intensive; however, it is an imperative tool for accurate measurements of the course’s impact.
The course focuses on skills such as teamwork, crew resource management and communication techniques. These skills together with proper preparation of the intubation equipment, having them organized in predetermined way and the use of cognitive aid have crucial effects on the success of safe intubation [13, 18, 19]. Similar results have been reported with training of otolaryngology residents on advanced airway skills .
The pediatric airway management course at CRESENT targets all pediatric residents. There are clear differences in the pre-test scores among the four levels of residents which give validity to the assessment tool used. However, the junior residents show the greatest improvement in their post-test scores to the level of the seniors which strengthens the effectiveness of the course. Training for airway management including endotracheal intubation should be conducted early during residency to get the maximum benefit . The importance of integrating airway management course into training programs for residents who manage critically ill children is essential as it reflects directly on patients’ outcome and safety [15, 17]. Attending advance life support courses once every two years is insufficient to improve intubation management . A simulation-based education curriculum for a residency program is best constructed in a modular fashion . A pediatric airway management course is one of these modules that best be administered early in the residency program.
The pediatric airway management simulation course at CRESENT is effective in improving the knowledge and practical skills of pediatric residents. Although the greatest improvement is noted among junior residents, learners from different residency levels have comparable knowledge and practical skills at the end of the course. Similar courses need to be integrated in the pediatric residency curriculum preferably at early stage of residency programs. Further research is needed to study skills’ retention and more importantly its impact on patients’ care. Our outcomes-based evaluation strategy has provided targeted insight to the strengths and areas for development in the course which we have acted upon.
The authors would like to acknowledge Myra L. Verano, CRESENT secretary.
Availability of data and materials
Data and materials are available on request.
SA, HM, and NFA made the study conception and design. NFA and SA carried out the acquisition of the data. SA, HM, NFA, and MSB participated in the analysis and interpretation of the data. SA and HL drafted the manuscript. SA and HL did the critical revision. All authors read and approved the final manuscript.
Ethics approval and consent to participate
The research was approved by IRB Committee at King Fahad Medical City (IRB 15-415).
Consent for publication
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.
- Reynolds SF, Heffner J. Airway management of the critically ill patient. Chest. 2005;I27(4):1397–412.Google Scholar
- Walz JM, Zayarumy M, Heard SO. Airway management in critical illness. Chest. 2007;13(2):608–20.View ArticleGoogle Scholar
- Jaber S, Amraoui J, Lefrant JY, et al. Clinical practice and risk factors for immediate complications of endotracheal intubation in the intensive care unit: a prospective, multiple-center study. Critical Care Med. 2006;34(9):2355–61.View ArticleGoogle Scholar
- Jabre P, Combes X, Lapostolle F, et al. Etomidate versus ketamine for rapid sequence intubation in acutely ill patients: a multicenter randomized controlled trial. Lancet. 2009;374(9686):293–300.View ArticlePubMedGoogle Scholar
- Samsoon GL, Young JR. Difficult tracheal intubation: a retrospective study. Anaesthesia. 1987;42(5):487–90.View ArticlePubMedGoogle Scholar
- Lee JH, Turner DA, Kamat P, Nett S, Shults J, Nadkarni VM, Nishisaki A. The number of tracheal intubation attempts matters! A prospective multi-institutional pediatric observational study. BMC Pediatr. 2016;16:58. doi:https://doi.org/10.1186/s12887-016-0593-y.View ArticlePubMedPubMed CentralGoogle Scholar
- Barch B, Rastatter J, Jagannathan N. Difficult pediatric airway management using the intubating laryngeal airway. Int J Pediatr Otorhinolaryngol. 2012;76(11):1579–82.View ArticlePubMedGoogle Scholar
- Sauer CW, Kong JY, Vaucher YE, Finer N, Proudfoot JA, Boutin MA, Leone TA. Intubation attempts increase the risk for severe intraventricular hemorrhage in preterm infants. A retrospective cohort study. J Pediatr. 2016;177:108–13.View ArticlePubMedGoogle Scholar
- Fitzpatrick J, Sanders J, Worthen B. Program evaluation: alternative approaches and practical guidelines. 4th ed. Upper Saddle River: NJ Pearson Education; 2011.Google Scholar
- Blanchard R, Torbeck L, Blondeau W. AM last page: a snapshot of three common program evaluation approaches for medical education. Acad Med. 2013;88(1):146.View ArticlePubMedGoogle Scholar
- Nestel D, Regan M, Vijayakumar P, Sunderji I, Haigh C, Smith C, Wright A. Implementation of a multi-level evaluation strategy: a case study on a program for international medical graduates. J Educ Eval Health Prof. 2011;3(13):1–15.Google Scholar
- Rudolph JW, Raemer DB, Simon R. Establishing a safe container for learning in simulation: the role of the presimulation briefing. Simul Healthc. 2014;9(6):339–49.View ArticlePubMedGoogle Scholar
- Ishizuka M, Rangarajan V, Sawyer TL, et al. The development of tracheal intubation proficiency outside the operating suite during pediatric critical care medicine fellowship training: a retrospective cohort study using cumulative sum analysis. Pediatr Crit Care Med. 2016;17(7):309–16.View ArticleGoogle Scholar
- Sudikoff SN, Overly FL, Shapiro MJ. High-fidelity medical simulation as a technique to improve pediatric residents’ emergency airway management and teamwork: a pilot study. Pediatr Emerg Care. 2009;25(10):651–6.View ArticlePubMedGoogle Scholar
- Leung SK, Cruz AT, Macias CG, Sirbaugh PE, Patel B. Improving pediatric emergency care by implementing an eligible learner endotracheal intubation policy. Pediatr Emerg Care. 2016;32(4):205–9.View ArticlePubMedGoogle Scholar
- Issenberg BS, McGaghie WC, Petrusa ER, Gordon DL, Scalese RJ. Features and uses of high-fidelity medical simulations that lead to effective learning: a BEME systematic review. Med Teach. 2005;27(1):10–28.View ArticlePubMedGoogle Scholar
- Mills DM, Wu CL, Williams DC, King L, Dobson JV. High-fidelity simulation enhances pediatric residents’ retention, knowledge, procedural proficiency, group resuscitation performance, and experience in pediatric resuscitation. Hosp Pediatr. 2013;3(3):266–75.View ArticlePubMedGoogle Scholar
- Long E, Fitzpatrick P, Cincotta DR, Grindlay J, Barrett MJ. A randomised controlled trial of cognitive aids for emergency airway equipment preparation in a Paediatric Emergency Department. Scand J Trauma Resusc Emerg Med. 2016;24:8. doi:https://doi.org/10.1186/s13049-016-0201-z.View ArticlePubMedPubMed CentralGoogle Scholar
- Malec JF, Torsher LC, Dunn WF, Wegmann DA, Amold JJ, Brown DA, Phatak V. The Mayo high performance teamwork scale: Reliability and validity for evaluating key crew resource management skills. Simul Healthc. 2007;2(1):4–10.View ArticlePubMedGoogle Scholar
- Amin MR, Friedmann DR. Simulation-based training in advanced airway skills in an otolaryngology residency program. Laryngoscope. 2013;123(3):629–34.View ArticlePubMedGoogle Scholar
- Makwana HD, Suthar NN, Gajjar MP, Thako AV. Developing competency in interns for endotracheal intubation: an educational article. Int J App Basic Med Res. 2016;6(3):201–4.View ArticleGoogle Scholar
- Deakin CD, Murphy D, Couzins M, Mason S. Does an advanced life support course give non-anaesthetists adequate skills to manage an airway? Resuscitation. 2010;81(5):530–43.View ArticleGoogle Scholar
- Lababidi H, Munshi F. Development of simulation curriculum in postgraduate programs. J Health Spec. 2015;3(1):17–21.View ArticleGoogle Scholar