Multiple testing has always been a fascinating topic for me. I still vividly remember the ``good old days’’ when I use Bonferroni corrections by default when testing multiple hypothesis simultaneously without understanding the rationale. This time I will try to facilitate my mastery of this topic by writing a blog about it.

Let’s say that we are conducting experiments to decide whether a two-sided coin is fair. To ensure we have a reasonable sample size, we flip the coin for 10 times. We may claim the coin to be unfair if all outcomes are head, becuase the probability of having 10 heads under the assumption of fair coin is \(\frac{1}{1024}\).

Assuming that are very diligent and decide to repeat our experiment for \(100\) runs, and conclude the coin to be unfair if at least one of our experiment run results in 10 heads. Under the assumption of fair coin, the probability of having 10 heads in at least one of our runs is equal to \(1 - (\frac{1023}{1024})^{100} \approx 0.093\). This implies that we will have about \(9\%\) chance to incorrectly conclude that a coin is not fair even though we use a very stringent rule for judging a coin to be unfair.

The approaches for handling this type of problem can be divided into two categories depending on their goal. Let’s denote the i-th null hypothesis as \(H_{0,i}\), and the i-th alternative hypothesis as \(H_{1,i}\), we have

• Family-wise error rate: \(FWER = P(\text{reject any} H_{0,i} | H_{0,i})\)
  • Bonferroni’s method
  • Siday’s method (only applicable to the case where the hypotheses are independent)
  • Holm’s method: stepwise method.
• False discovery rate: the expected value of the number of false rejections over the number of rejections.
  • Benjamini-Hochberg Method.