AN INDEPENDENT PUBLIC HEALTH READING ROOM
Places. People.
Possibilities.

A Small P Value Does Not Tell You How Large the Health Benefit Is

Paper collage of a notebook, tablet, magnifying glass and source card.
AI-created conceptual illustration of a fictional setting; not a record of people or outcomes.

A headline may call a result statistically significant without saying how much changed. That leaves out information a reader needs to understand whether the finding could matter in everyday life.

The Cochrane Handbook cautions against relying on a fixed P-value threshold to interpret findings. A P value addresses compatibility with a specified statistical model and null hypothesis; it is not the size of the effect.

Consider a fictional study of a score measured on a hundred-point scale. Knowing that its analysis reports a small P value does not tell you whether the difference was a fraction of a point or several points. You need the estimated difference and its uncertainty.

It also does not give the probability that the study’s conclusion is true, or that the result happened “by chance” in a simple everyday sense. Such shortcuts replace a conditional statistical statement with a much broader claim.

Use a short reading sequence: identify the comparison, find the outcome and units, record the effect estimate, then read the confidence interval and the analysis methods. Check whether several outcomes or analyses were explored, because the study’s overall design matters to interpretation.

Do not reverse the mistake when a result crosses an arbitrary threshold in the other direction. A result labelled non-significant does not automatically prove absence of a worthwhile effect; an imprecise study can leave substantial uncertainty.

For a public-facing summary, describe the magnitude and limits in ordinary language before using a statistical label. This keeps the discussion focused on the actual question and evidence rather than treating one small number as a certificate of practical importance. Personal healthcare decisions still require appropriate clinical context beyond this reading exercise.

Editorial illustration from the site library.