EVALUATION OF A CUMULATIVE EXIT-FROM-DEGREE OBJECTIVE STRUCTURED CLINICAL EXAMINATION (OSCE) IN A GULF CONTEXT

Abstract

This study aimed to evaluate the psychometric properties of the 2nd iteration of an Objective Structured Clinical Examination (OSCE) for graduating pharmacy students in Qatar. A secondary objective of this study was to identify quality improvement opportunities for design, implementation, and evaluation of the OSCE. The psychometric analyses occurred as follows: Cut score determination using borderline regression method; predictive validity using regression and correlation of select course grades and assessments with OSCE scores, concurrent validity using correlation between other cumulative assessments and OSCE scores, risk of bias using correlation between assessors’ analytical and global scoring, content validity using student-feedback forms, and interrater reliability using intra-class correlation coefficients (ICCs), and internal consistency using Cronbach’s alpha. Pearson and Spearman correlation statistics were conducted at α level < 0.05. A series of two focus groups and subsequent qualitative content analysis were conducted with key stakeholders to identify strengths, weaknesses, opportunities, and challenges regarding OSCE implementation. Total cut score for the exam was 55.3%. Overall pass rate was 79.2%. OSCE scores correlated moderate-strongly with course grades of Professional Skills and Integrated Case-based Learning, and formative OSCE assessments. Course grades for medicinal chemistry were not correlated with OSCE scores. OSCE scores were moderately predicted by Professional skills course grades (52.3%) and its formative OSCE assessment (61.2%). Average correlation between analytical and global grades for all assessors was 0.52. A total of 90% of the stations were deemed to reflect practice, according to student perceptions. The average intraclass correlation coefficient for analytical checklists scores, global scores, and total scores were 0.88 (0.71 – 0.95), 0.61 (0.19 – 0.82), and 0.75 (0.45 – 0.88) respectively. Cronbach’s alpha of students’ performance in global scores across stations was 0.87, and 0.93 in terms of total scores. Focus groups confirmed content validity as a weakness yet spoke to training and assessment techniques as both strengths and areas for improvement. In sum, the 2nd iteration of a cumulative OSCE for graduating pharmacy students in Qatar was deemed valid and reliable, however refinements can be implemented in future iterations to further improve the exam as a high stakes assessment.