Inter-school variations in the standard of examiners’ graduation-level OSCE judgements

Yeates, Peter, Maluf, Adriano, McCray, Gareth, Kinston, Ruth, Cope, Natalie, Cullen, Kathy, O’Neill, Vikki, Cole, Aidan, Chung, Ching-wa, Goodfellow, Rhian, Vallender, Rebecca, Ensaff, Sue, Goddard-Fuller, Rikki and McKinley, Robert 2025. Inter-school variations in the standard of examiners’ graduation-level OSCE judgements. Medical Teacher 47 (4) , pp. 735-743. 10.1080/0142159x.2024.2372087

Preview

PDF - Published Version Available under License Creative Commons Attribution Non-commercial No Derivatives. Download (2MB) | Preview

Abstract

Introduction Ensuring equivalence in high-stakes performance exams is important for patient safety and candidate fairness. We compared inter-school examiner differences within a shared OSCE and resulting impact on students’ pass/fail categorisation. Methods The same 6 station formative OSCE ran asynchronously in 4 medical schools, with 2 parallel circuits/school. We compared examiners’ judgements using Video-based Examiner Score Comparison and Adjustment (VESCA): examiners scored station-specific comparator videos in addition to ‘live’ student performances, enabling 1/controlled score comparisons by a/examiner-cohorts and b/schools and 2/data linkage to adjust for the influence of examiner-cohorts. We calculated score impact and change in pass/fail categorisation by school. Results On controlled video-based comparisons, inter-school variations in examiners’ scoring (16.3%) were nearly double within-school variations (8.8%). Students’ scores received a median adjustment of 5.26% (IQR 2.87–7.17%). The impact of adjusting for examiner differences on students’ pass/fail categorisation varied by school, with adjustment reducing failure rate from 39.13% to 8.70% (school 2) whilst increasing failure from 0.00% to 21.74% (school 4). Discussion Whilst the formative context may partly account for differences, these findings query whether variations may exist between medical schools in examiners’ judgements. This may benefit from systematic appraisal to safeguard equivalence. VESCA provided a viable method for comparisons.

Item Type:	Article
Date Type:	Publication
Status:	Published
Schools:	Schools > Medicine
Additional Information:	License information from Publisher: LICENSE 1: URL: http://creativecommons.org/licenses/by-nc-nd/4.0/, Start Date: 2024-07-08
Publisher:	Taylor and Francis Group
ISSN:	0142-159X
Date of First Compliant Deposit:	26 March 2025
Date of Acceptance:	20 June 2024
Last Modified:	26 Mar 2025 12:11
URI:	https://orca.cardiff.ac.uk/id/eprint/177175

Actions (repository staff only)

Edit Item