BiostatisticsTier 1Medical Sciences concept

P-value - definition and common misinterpretations

Core concept

p = P(data this extreme or more extreme | H0 true)
- Conditional on the null being true - it says nothing about whether the null is true
p < 0.05 -> reject H0. That threshold is arbitrary convention, not a law

What p is NOT

Misreading	Why wrong
"5% chance the null is true"	That is P(H0 \	data) - needs a prior, i.e. Bayes
"95% chance the result is real"	Same inversion
"5% chance the result is due to chance"	The calculation already assumes chance alone
"p = 0.06 means no effect"	Non-significant, not null. Absence of evidence is not evidence of absence
"p = 0.001 means a big effect"	p conflates effect size and sample size - a trivial effect in n=100,000 gives a tiny p
"p = 0.04 and p = 0.06 differ"	The difference between significant and non-significant is itself usually not significant
"Two trials, one p=0.04 one p=0.20, so they conflict"	Compare the CIs, not the p-values

The confidence interval carries everything p does, plus the effect size and its precision
- If a 95% CI for a difference excludes 0 (or for a ratio excludes 1), p < 0.05 - they are the same test
- Report the CI
Fragility index - how many outcome events would have to switch to lose significance; many "positive" trials have an index of 1-3
p-hacking / multiplicity - 20 independent tests at alpha 0.05 -> ~64% chance of at least one spurious positive
- Guards: pre-registration, pre-specified primary endpoint, hierarchical testing, Bonferroni (alpha/k)

Statistically significant is not clinically significant - always convert to an absolute measure
- RRR sounds large, ARR and NNT = 1/ARR are what the patient experiences
- A 30% RRR on a 1% baseline risk is an ARR of 0.3% - NNT 333
Surrogate endpoints can move with a small p and no clinical benefit (CAST: flecainide suppressed ectopy and increased mortality)
A wide CI around a null result means the trial was inconclusive, not negative - the correct answer to a negative underpowered trial is "we do not know"
Composite endpoints reach significance on their softest, commonest component

3 more sections, plus exam facts

Premium unlocks every note across every specialty, and the full exam fact library behind it.