Table 1 Considerations for implementing a summative assessment with simulation

Competency to be assessed

Clear definition of competency

Know how to act in a professional situation

Identify internal resources: knowledge, skills, behavior, and reasoning

Identify external resources: equipment, written or electronic resources), colleagues, and so on to mobilize

The practitioner is able to handle an in-hospital cardiac arrest (CA)

ACLS algorithm, airway management, leadership, management according to the type of CA (e.g., asystole, pulseless electrical activity, ventricular fibrillation)

e.g., defibrillator, cognitive aids (a chart, a checklist, …), ECMO team, …

Number of competencies

Consider the possibility of assessing one or more competencies simultaneously

In-hospital CA alone, or CA in adult patient and/or in specific conditions (e.g., child, pregnant, …)

Measurements

Consider measuring performance in representative and diverse situations

CA in a young polytrauma patient, in an elderly diabetic patient, in a pregnant woman or in a child out-of-hospital

Assessment

Context authenticity

Complex problems

Adapt the complexity to the training level

Ensure context relevance to future or current professional practice

Interprofessional situations (vs uniprofessional)

e.g., CA due to hyperkaliemia in a patient with renal failure

Complexity may be tuned for an expert with patient's chronic use of beta-blockers

CA occurs in an ambulance or in an emergency room or in OR or in ICU

Prefer a situation where the learner is not alone such as a member of an emergency team and not as a first responder in the street

Standardization

Tasks and requirements known before by the participants

Direct observation associated with a phase of student interaction (questioning)

Rate with a checklist or a rubric

Send to the learner the assessment template prior to the assessment

The simulation is followed by a debriefing (feedback)

Correction criteria

Multiple sources and/or iteration (e.g., repeated performances of the same scenario)

Clear and specific objectives

Adjusted to the assessed knowledge or to the simulation

Integration of self-assessment

Consider only important errors

Strategies (cognitive and metacognitive) assessed during the interaction phase

Prior consensus on rating and definition regarding expected level of development

e.g., time from the start of VF to the first external electric shock and/or compliance with ACLS steps and/or quality of external cardiac massage (visual and/or via sensors)

Only items that have been previously decided are assessed (see above)

It is not possible to assess the use of the defibrillator if the situation is pulseless electrical activity

6 instead of 5 min between 2 doses of adrenaline (minor error) versus no recognition of a shockable rhythm (major error)

Ask questions during feedback phase: “Can you remind me of the administration schedule for epinephrine in CA?” (cognition). “I have observed that you administered it every minute, but as you have just said and as I think it is every 3 to 5 min, could you explain why in the situation you administered it every minute?” (metacognition)

Identify minor and major errors together (all instructors involved in the assessment of this competency). Define the number of acceptable minor and/or major errors to validate the acquisition or not of the competency at this level of development

Scenarios

Development

Developing scenarios only after defining the skills and or competences to be assessed

Ensuring the scenario reflects professional reality

Incorporating the targeted skills into a scenario representing professional practice, rather than a task trainer, for example

e.g., if we want to evaluate the use of the defibrillator, we need to construct a scenario where the patient has VF or VT

e.g., use a hyperkalemia CA scenario after a burial extraction but not when releasing a tourniquet after a knee replacement for an emergency physician

Prefer to use a scenario with a clinical history of CA to assess CPR skills rather than performing CPR in a skill station

Multiple skills

Several stations with short scenarios (e.g., 5–6 min) each are preferable to long scenarios (e.g., > 20 min)

Critical situation

Ensure that all steps can be assessed. E.g., the use of ECMO is reserved for refractory CA and cannot be considered if the scenario lasts for 5 min and begins with the recognition of the arrest. In this case, a scenario with a CA that has already been under management for 15 min should be used

Test prior to use

Validity, reliability, reproducibility

The scenarios used should be pre-tested by the teaching team including using the assessment forms

Simulators (High and low-Technology)

Use and difficulty level validated

e.g., if intubation is expected during the scenario, the chosen manikin should allow it

Assessment test

standardization

(Fairness)

Facilitator's role and intervention specified in advance

Only one candidate per station

What can the facilitator do? E.g., can he/she guide on 4H-4 T if the learner does not think about it?

Practical conditions

Minimum number of scenarios (8 to 15) [157]

Incentive to verbalize after action

(Reasoning, what is done or not done)

Scenarios in different circumstances (in and out-of-hospital), different causes (4H-4 T), different ages (child to elderly adult)

To be recalled in the pre-briefing

Raters

At least, two raters

Ideally, a rater should be involved in the formative assessment program

e.g., clinical supervisor, ACLS instructor, simulation instructor who has supervised the learner during the formative sessions, …