Table 2 Example of the development of a tool to assess technical skill achievement in a simulated situation, based on work by Oriot et al., Downing, and Messick’s framework [38, 46, 47]
From: Simulation-based summative assessment in healthcare: an overview of key principles for practice
Source of validity | Method | Judgment criteria | Results |
|---|---|---|---|
content | 1. Description of the checklist development by 2 experts 2. Review by 2 outside experts 3. Definitive Checklist | Relevance of items Adapted illustration of the skill Conditions of skill achievement | Obtaining a list of 12 items (after the initial proposal of 20 items) |
Response process | Pilot study, search for error sources Adapting items Defining units of measurement | Interrater reproducibility Item content (redundant, inaccurate) Controlling the sources of measurement errors Weighing items | Fusion/removal of redundant items Minutes, degrees, centimeters justification |
Internal structure | Internal coherence Reproducibility Discrimination of learners | Cronbach Alpha Coefficient, interrater: Cohen Kappa, ICC | Cronbach result Correlation between 2 raters |
Comparison with other variables | Score vs success or failure of the procedure Score vs theoretical assessment Score vs previous experience/level of expertise | Correlation between procedure success or theoretical assessment and score with the tool | Time for success, score for success and rating |
Consequences | Minimum passing score | Pass-fail score with procedure success | 14/20 |