P-value driven methods were underpowered to detect publication bias: analysis of Cochrane review meta-analyses

Abstract

To investigate the effect of number of studies in a meta-analysis on the detection of publication bias using p-value driven methods. The proportion of meta-analyses detected by Egger's, Harbord's, Peters', and Begg's tests to have asymmetry suggestive of publication bias were examined in 5014 meta-analyses from Cochrane reviews. P-values were also assessed in meta-analyses with varying number of studies, while symmetry was held constant. A simulation study was conducted to investigate if the above tests underestimate or overestimate the presence of publication bias. The proportion of meta-analyses detected as asymmetrical via Egger's, Harbord's, Peters', and Begg's tests decreased by 42.6%, 41.1%, 29.3%, and 28.3%, when the median number of studies in the meta-analysis decreased from 87 to 14. P-values decreased as the number of studies increased in the meta-analysis, despite the level of symmetry remaining constant. The simulation study confirmed that when publication bias is present, p-value tests underestimate the presence of publication bias, particularly when study numbers are small. P-value based tests used for detection of publication bias related asymmetry in meta-analysis require careful examination as they underestimate asymmetry. Alternative methods not dependent on the number of studies are preferable.