Comment on a review of methods to assess publication and other reporting biases in meta-analysis

Abstract

We read with interest the well-written review on the assessment of publication and other reporting biases in meta-analysis by Page et al.1 in which both funnel plots and tests for its asymmetry were described. The decision on funnel plot asymmetry is usually based on visual inspection and/or a quantitative assessment of the degree of (a)symmetry. All tests described for binary outcomes (i.e., Egger, Begg and Mazumdar, Harbord, and Peters tests) are p-value driven, where a p-value of <0.1 is usually taken to be indicative of asymmetry. One of the drawbacks with these p-value driven methods is that they are underpowered when the number of studies is small and decisions on asymmetry are dependent on the number of studies included.2

This is particularly problematic because, for the same level of (a)symmetry, a test could produce discordant results. In a meta-analysis with 11 trials investigating the effect of metformin versus usual care/placebo in the incidence of gestational diabetes mellitus,3 Egger's regression (p-value = 0.163) did not indicate asymmetry of the funnel plot at the usual threshold. When each study was added to the dataset twice and thrice; therefore, leaving the level of (a)symmetry unchanged, but with increased study numbers, the p-values of Egger's regression for the funnel plots with 22 and 33 trials dropped to 0.035 and 0.008, suggesting asymmetry of the plots