Skip to main content

Table 2 The five sources of evidence validity framework

From: Validation of educational assessments: a primer for simulation and beyond

Source of evidence Definition Examples of evidence
Content “The relationship between the content of a test and the construct it is intended to measure” [24] Procedures for item sampling, development, and scoring (e.g., expert panel, previously described instrument, test blueprint, and pilot testing and revision)
Internal structure Relationship among data items within the assessment and how these relate to the overarching construct Internal consistency reliability
Interrater reliability
Factor analysis
Test item statistics
Relationships with other variables “Degree to which these relationships are consistent with the construct underlying the proposed test score interpretations” [24] Correlation with tests measuring similar constructs
Correlation (or lack thereof) with tests measuring different constructs
Expert-novice comparisons
Response process “The fit between the construct and the detailed nature of performance . . . actually engaged in” [24] Analysis of examinees’ or raters’ thoughts or actions during assessment (e.g., think-aloud protocol)
Assessment security (e.g., prevention of cheating)
Quality control (e.g., video capture)
Rater training
Consequences “The impact, beneficial or harmful and intended or unintended, of assessment” [27] Impact on examinee performance (e.g., downstream effects on board scores, graduation rates, clinical performance, patient safety)
Other examinee effects (e.g., test preparation, length of training, stress, anxiety)
Definition of pass/fail standard
  1. See the following for further details and examples [20, 25, 26]