Application of a systematic approach to evaluating psychometric properties of a cumulative exit-from-degree objective structured clinical examination (OSCE)

Abstract

Background and purpose Objective structured clinical examinations (OSCEs) are considered gold standard performance-based assessments yet comprehensive evaluation data is currently lacking. The objective of this study was to critically evaluate the psychometric properties of a cumulative OSCE for graduating pharmacy students in Qatar for which policies and procedures were adapted from a Canadian context.

Educational activity and setting A 10-station OSCE was conducted for graduating students in Qatar. Evaluation included assessment of pass rates, predictive validity, concurrent validity, internal validity, content validity, interrater reliability, and internal consistency.

Findings Twenty-six students completed the OSCE. Three stations achieved pass rates < 80%. Scores from professional skills and case-based learning courses, formative OSCEs, and cumulative grade point averages correlated with OSCE scores (p < 0.05). Average correlation between assessors’ analytical and global scoring was moderate (r = 0.52). Average interrater reliability was excellent for analytical scoring (ICC = 0.88) and moderate for global scoring (ICC = 0.61). Excellent internal consistency was demonstrated for overall performance (α = 0.927). Students generally agreed stations represented real practice scenarios (range per station, 30–100%).

Discussion and summary The evaluation model identified strengths and weaknesses in assessment and curricular considerations. The OSCE demonstrated acceptable validity and reliability as an adapted assessment.