Mental effort in the assessment of critical reflection: Implications for assessment quality and scoring

Abstract

Introduction: Critical reflection is a mainstay in the training of health professionals, yet assessment of reflection is commonly described as difficult, taxing, and resulting in inconsistent scoring across assessors. At the same time, there is evidence from experiential and simulation settings that assessors' mental effort may explain assessor variability, which could be a target for simplifications in assessment design. Assessors' mental effort for assessment of reflection is currently unknown. This study aimed to determine reliability of rubric scoring of critical reflection, variation in pass-fail rates, and the relationship between reflection scores and assessors' perceived mental effort. Methods: Eleven assessors were recruited to assess six reflection assignments using a published rubric. Mental effort was measured using the Paas scale for each assignment assessed and was correlated with rubric scores for each assignment. Results: Findings showed inconsistency in scoring between assessors, resulting in varying pass rates for each assignment (55–100%). All assignments demonstrated negative correlations between rubric scores and perceived mental effort (r = −0.115 to −0.649). Conclusions: Findings support the notion that more work should be done to optimize assessment of critical reflection. Future studies should focus on disentangling the influence on mental effort of scoring tools, assignment structures, and writing quality.