When conducting multiple safety tests, what is a key caveat about p-values, and what is a common remedy?

• A. Small p-values are always reliable regardless of the number of tests.
• B. Small p-values can occur by chance when many tests are performed; apply multiple-comparison corrections such as Bonferroni.
• C. P-values replace the need for replication.
• D. P-values determine study duration.

Answer: (B)

When you run many safety tests, small p-values can appear just by chance. Each test has a likelihood of a false positive, and with a large number of tests the probability that at least one test looks significant purely by luck grows. This can lead you to chase spurious findings if you don’t adjust for the sheer number of tests.

A common remedy is to apply multiple-comparison corrections to control how often you declare significance across all tests. Bonferroni is the simplest approach: you take your desired overall significance level and divide it by the number of tests, then only call a result significant if its p-value is below that stricter threshold. This helps keep the chance of any false positive across the whole set in check. There are more powerful methods, like Holm or Benjamini-Hochberg, that balance false positives with discovery, but the core idea is the same: adjust the significance criterion when many tests are in play. Replication and focusing on effect sizes also help ensure findings are robust beyond p-values.