Table 1 Development of the Night-onCall (NOC) event

Clinical cases/mixed modality

Case 1: Oliguria “I am calling about Mr. Jackson, 64-year-old man S/P Elective endovascular repair of AAA, post-operative day 3. His urine output has dropped and he has mild abdominal pain” (Has urinary retention, BPH).

WISE-onCall module with 3 practice cases.

Case 2: Oliguria “I am calling about Mr. Taylor. 57-year-old man here for observation to rule out acute cardiac ischemia and pulmonary embolism. His urine output has dropped and he remains without chest pain” (received contrast for a cat scan).

Case 1: Oliguria (same).

WISE-onCall module with three practice cases.

Case 2: Oliguria (same).

Case 3: headache “I’m calling about Mr. Johnson, 64-year-old man S/P a AAA repair day 3, and is complaining of a severe headache” (has a blood pressure of 195/99 and a history of HTN).

Case 4: headache: “Hi. Are you covering for Mr. Kolinsky, 62-year-old man S/P internal fixation of an ankle fracture…I wanted to let you know that he is having a severe headache” (history of migraines on propranolol for prevention).

Form clinical question and retrieve evidence to advance clinical care.

Culture of safety analysis of a paper case: vignette describing pre-entrustable peer on internal medicine clerkship-structured response identifying evidence of behaviors and assessment of entrustment.

Handoff all two cases to fellow intern (standardized): prioritize based on urgency. Assessment of entrustment.

Case 1: Oliguria (same).

Plus: oral presentation to attending.

WISE-onCall module with three practice cases.

Case 2: Oliguria (same)

Case 3r: headache (revised) “Hi. Are you covering for Mr. Brooks, 60-year-old man being treated for Diverticulosis, …I wanted to let you know his blood pressure is really high” (195/99 currently, non-focal neuro-exam, history of migraine headaches on propranolol for prevention)?

Form clinical question and retrieve evidence to advance clinical care (same).

Case 4r: Go “get” consent (revised): “Hi. This is Randy, your second year resident. You are covering Mr. Smith a 40 y/o with a cough, fever and pleural effusion. You need to go consent him for a thoracentesis. I will meet you at the bedside in 1 h.” (The resident will explain the procedure if asked, patient’s husband is in the room).

Culture of safety analysis of a paper case: same.

Handoff all four cases to fellow intern (standardized): prioritize based on urgency. Assessment of entrustment.

Number and types of participants

52 4th-year graduating medical students.

66 4th-year graduating students.

42 3rd-year students (rising seniors).

89 students.

35 4th-year, 12 3rd-year accelerated, 36 3rd-year, 65-year pathway.

Event length

3 h/student, Over 3 full days in simulation center.

3 h/student, over 9 full days in simulation center.

4 h/student, over 16 half days in simulation center.

Incentive

$100/Student, IRB-approved protocol.

$100/Student, IRB-approved protocol.

$100/student, IRB-approved protocol.

EPA’s addressed and assessed

1–5, 9,12

Piloted oral presentation, handoff, evidence-based medicine, culture of safety.

1–10, 12–13

1–13

Study questions

In what ways are our near graduates ready for internship?

Does WISE-OnCall “just in time” improve core clinical skills required for common clinical coverage issues?

Do different forms of feedback (short-form checklist vs. whole-form checklist) provided during the practice cases have an impact on learning outcomes?

Does simulated clinical exposure before WISE-onCall enhance learning from it?

Does WISE-onCall improve clinical performance in content discordant cases?

Exploratory: 3rd-year vs. 4th-year students?

Which core EPAs for entering residency can we reliably assess in an integrated authentic simulated experience?

Is it feasible to assess all core EPAs for entering residency in an integrated authentic simulated experience?

What are the differences in readiness for residency among clinically experienced students in different curricular pathways?

Measurements (assessor: assessed domains)

SP: Communication skills (data gathering, rapport building, patient education and counseling), history gathered, physical exam, professionalism, recommendations (entrustment equivalent).

SN: collaboration, inter-professional communication, rapport building, professionalism (entrustment).

Patient note: reporter, interpreter, manager, clinical reasoning.

Faculty: clinical reasoning, entrustment.

Structured domain specific medical knowledge (clinical schema).

Plus

Paper case: culture of safety: entrustment of peers.

Medical librarian: ability to formulate answerable clinical questions and identify a literature based answer.

Peer: handoff quality and/entrustment.

Plus

Faculty: oral presentation skills and entrustment.

Case no. 3: SP/SN: recognize a patient requiring urgent or emergent care and initiate evaluation and management.

Case no. 4: SP/standardized resident/spouse: ability to perform an ethical and legal informed consent discussion and effectively include family members.

Feedback, findings and remaining questions

• Students appreciate the opportunity to practice and learn before July 1.

• WISE-onCall module is useful “just in time”.

• Students question authenticity of working with nurse in patient’s room.

• Extreme variability in measured “readiness” and sophistication in clinical schema.

• Majority of students improved significantly after WISE-onCall (some did not). Does this reflect readiness for learning from clinical cases?

• Simpler forms of feedback with in Wise onCall are as effective as more complex ones (RCT).

• Although they need to be refined, our assessments were reasonably reliable, authentic and synthetic.

• Many of the common topics required for transition from UME to GME can be assessed and addressed using this style of blended assessment/learning experiences.

• All clinical students (3rd and 4th) appreciate the practice and authenticity.

• Educational utility is high.

• Students demonstrate the best clinical skills and clinical reasoning after they complete an SP/SN case on the same topic before a Wise onCall module (neither alone is enough).

• MS 3s have more comprehensive basic clinical skills than MS 4s.

• Both MS 3s and 4s get a significant boost in content specific structured knowledge from blended WISE-OnCall and simulation experience.

• 4th-year students gained more in the domains of clinical management and overall clinical reasoning than the 3rd-year students.

• This may be secondary to boosting effect of the experience on knowledge and skills they had obtained but forgot.

• Almost all students recognize pre-entrustable “culture of safety” behaviors in a peer and can recommend strategies to address these.

• The quality of ability to formulate answerable clinical questions and identify a literature-based answer is highly variable.

• Continued enthusiasm for high educational yield of the event.

• Feasible to assess all 13 core EPAERs confirmed.

• No significant differences among students in accelerated MD program and traditional 4th-year program (small sample).

• Attendings impressed with variability in intern readiness based on oral presentation.

• Both competency measures and entrustment measures can be made.

• What should we do with students who perform poorly on NOC?

• What would be the more useful design for educational handoffs from UME to GME?

• Can we establish predictive models and cut offs for the data produced in NOC?