Do-it-yourself meta-analysis

Dynomight has looked at the health effects of vitamin D supplementation. The large-scale meta-analyses that have been performed conclude there is no significant effect, even though individual studies relatively consistently point in the direction of an effect. This means our failure to detect a significant effect may be due to low experiment power.Dynomight suggests a mechanism for that: what if the low power of the existing randomised trials is because they tend to be run in countries that fortify common foods with vitamin D? That would mean, practically speaking, all arms of the trial get the treatment. To investigate, Dynomight pulls out a table with the results only from studies where participants entered with relatively low values of vitamin D in their blood.TrialAll-cause mortalityTrivedi0.90 (0.77 to 1.07)whi0.92 (0.83 to 1.01)Lyons0.99 (0.93 to 1.05)record0.93 (0.85 to 1.02)The all-cause mortality is reported in odds ratios, which is a multiplicative scale. If the number is less than one, it means vitamin D supplementation was found to reduce all-cause mortality. But the 95 % confidence intervals in parentheses all straddle 1, meaning none of the studies were able to show a significant effect at that confidence level.Unfortunately, no formal meta-analysis has been done on this specific subset of studies. But we can make a quick and dirty one!Sign test (counting coinflips)We can tell immediately from the table that four out of four studies have a number less than one, i.e. they show a beneficial effect of vitamin D supplementation.This is a primitive form of meta-analysis! We count the total number of studies, and how many of them support the hypothesis. If we assume no effect, then half the trials should show a benefit, and half should show harm. What are the chances of flipping a coin four times and getting the same result on all four of them? 12.5 %.Thus, the pooled p-value of the combined trials, when we look only at the direction of their result, is 12.5