Multiple comparisons problem

Multiple comparisons, multiplicity or multiple testing problem occurs when many statistical tests are performed on the same dataset. Each test has its own chance of a Type I error (false positive), so the overall probability of making at least one false positive increases as the number of tests grows. In statistics, this occurs when one simultaneously considers a set of statistical inferences or estimates a subset of selected parameters based on observed values. The probability of false positives is measured through the family-wise error rate (FWER). The larger the number of inferences made in a series of tests, the more likely erroneous inferences become. Several statistical techniques have been developed to compensate for the number of inferences being made—for example, by requiring a stricter significance threshold for individual comparisons.