Overconfident results with the bivariate random effects model for meta-analysis of diagnostic accuracy studies

Abstract

Meta-analyses of diagnostic accuracy studies are a fundamental component of evidence-based medicine, and they are extensively used in medical imaging and the clinical laboratory. Techniques specifically developed to combine independent studies of diagnostic accuracy and provide pooled estimates for sensitivity (Se), specificity (Sp), positive (pLR) and negative (nLR) likelihood ratios are relatively new. In 2001, Rutter and Gatsonis proposed the hierarchical summary receiver operating characteristic (HSROC) model,1 and in 2004 Macaskill described an empirical Bayes approach.2 Soon after, in 2005, Reitsma et al. proposed the bivariate random effects model,3 which has been widely adopted and is the most commonly used method for diagnostic meta-analysis.4

However, as pointed out by Diaz,5 the statistical performance of the bivariate model has not been scrutinized. Diaz found that the performance of the bivariate model deteriorates when between-study heterogeneity increases and the number of studies decrease.5 Our simulation studies found similar results—with moderate levels of heterogeneity (tau2 = 1), the coverage probabilities of Se, Sp, and the diagnostic odds ratio (DOR) with the bivariate model dropped below the nominal level.6 Diagnostic accuracy studies usually favor sensitivity over specificity, or vice versa leading to diagnostic 2 × 2 tables with one or more of the cells with low frequency or zero counts. Thus, extreme DORs are more commonly observed in diagnostic than in intervention meta-analyses, which leads to high levels of heterogeneity (despite the wide confidence intervals of the studies).7