Measures of effect - p-values

Definition

A p-value is the probability of observing a result at least as extreme as the one obtained, assuming the null hypothesis (no true effect) is actually true - conventionally, a p-value below 0.05 is considered "statistically significant," though this threshold is an arbitrary convention, and statistical significance does not automatically equal clinical importance
Reported in virtually every clinical research study, but widely misinterpreted, making correct understanding of what a p-value actually does and does not convey a foundational critical appraisal skill

A p-value does not indicate the probability that the null hypothesis is true, nor the probability that the observed effect is real, nor the magnitude/clinical importance of an effect - it only indicates how compatible the observed data is with the null hypothesis being true, a specific and limited statistical statement frequently misrepresented in casual interpretation as answering a broader question than it actually does

A very large study can find a p-value below 0.05 for a clinically trivial, unimportant effect size simply because large sample size gives the statistical power to detect even very small true effects - statistical significance and clinical significance are separate considerations, and a significant p-value should always be interpreted alongside the actual effect size (and its confidence interval) rather than treated as sufficient on its own
Multiple comparisons (testing many hypotheses within a single study) inflate the chance of finding at least one significant result purely by chance, requiring appropriate statistical correction when many comparisons are performed

P-values should never be interpreted in isolation - always alongside the actual effect size and its confidence interval, and considering the clinical relevance of the magnitude of effect found, not just whether the threshold was crossed
Awareness of multiple comparison problems is important when a study reports many p-values across numerous outcomes/subgroups, given the elevated chance finding rate this creates

Statistical vs clinical significance distinction (a frequently tested, foundational point), multiple comparisons problem and false-positive inflation, large sample sizes detecting clinically trivial effects as statistically significant, common misinterpretation of what a p-value actually represents
A p-value below 0.05 in a very large study can reflect a genuinely trivial, clinically unimportant effect size, since large sample sizes provide the statistical power to detect even tiny true effects as "significant" - this is why statistical significance should never be treated as equivalent to clinical importance, and why the actual effect size (not just whether p<0.05) should always be examined when interpreting a study's clinical relevance

4 more sections, plus exam facts

Premium unlocks every note across every specialty, and the full exam fact library behind it.