Factors of the Asymmetric Non-Uniform DIF Detection Rate When Using the Alternative Mantel-Haenszel Procedure

Abstract

Test-item bias has become an increasingly challenging investigation in statistics and education. A popular method, the Mantel-Haenszel (MH) Test, is used for detecting non-uniform differential item functioning (DIF) but requires constructing several performance tiers to maintain robustness. Within the last two decades, the Alternative Mantel-Haeszel (AMH) Test (1994) was developed as a proxy procedure requiring only two scoring tiers. However, there is little information on how important factors like comparison group sizes, difficulty of questions, or question discrimination affect its ability to detect bias. In this statistical study, we investigate how item difficulty and discrimination as well as the ratio between the focal and reference groups examined impact the likelihood of the AMH detecting DIF. This research begins with a simulation phase, in which test scores are generated under three conditions: three commonly-used difficulty levels (easy, medium, and hard), two discrimination levels (referred to as 'low' and 'high'), and three group comparison ratios (1:1, 2:1, and 5:1). From the simulation, the detection rates of the AMH Test are compared to those of another common test, like the Breslow-Day (BD) Test. The results are then used as input to fit post-hoc statistical models to determine, which of the three factors affect AMH detection behavior. The study concludes with an application involving college-level test data comparing students across gender, nationality, and meta-major.