Measures of effect - type 1 and type 2 errors

The 2x2

	Truth: null is TRUE	Truth: null is FALSE
Reject the null	Type I error (alpha) - false positive	Correct - power (1 - beta)
Do not reject	Correct	Type II error (beta) - false negative

A type I (alpha) error is the incorrect rejection of a true null hypothesis - a false-positive finding - with its probability set by the chosen significance level
- Conventionally alpha = 0.05
Type II (beta) error - failure to reject a false null hypothesis; a false negative
- Conventionally beta = 0.20, giving 80% power

Mnemonic: type I = seeing something that isn't there (alpha = "gullible"); type II = missing something that is (beta = "blind")

Most negative trials in medicine are underpowered rather than truly null - type II error is the commoner error in the literature, while type I error dominates in high-throughput and subgroup analyses

The probability of observing a result at least as extreme as the one obtained, IF the null hypothesis were true
It is NOT
- the probability that the null hypothesis is true
- the probability that the result occurred by chance
- a measure of effect size or clinical importance

Power increases with:

Larger sample size - the only factor the investigator freely controls
Larger true effect size
Smaller variance (more precise measurement, more homogeneous population)
Higher event rate in the control arm
Larger alpha (trading type II error for type I)
Continuous rather than dichotomised outcomes; paired rather than unpaired designs

With alpha = 0.05, testing 20 independent hypotheses gives a ~64% chance of at least one false positive (1 - 0.95^20)
Corrections
- Bonferroni - divide alpha by the number of tests. Simple, conservative, loses power
- Holm-Bonferroni, Hochberg - stepwise, less conservative
- False discovery rate (Benjamini-Hochberg) - controls the proportion of false positives among rejections; standard in genomics
- Alpha-spending functions (O'Brien-Fleming, Pocock) for interim analyses

Suspect a type I error when
- The finding is a secondary or subgroup outcome
- Many comparisons were performed without correction
- The p value sits just under 0.05
- The result is biologically implausible or contradicts a larger body of evidence
- The trial was stopped early at an interim analysis
Suspect a type II error when
- A "negative" trial has a wide confidence interval that includes a clinically important effect
- The observed event rate was much lower than assumed in the sample size calculation
- Recruitment fell short of target

A 95% CI excluding the null is equivalent to p < 0.05, but it also shows the range of effects compatible with the data
"Negative" trial with a CI of 0.60 to 1.05: this is not evidence of no benefit

Pre-specify a single primary outcome and analysis plan; register the protocol before enrolment
Adjust for multiplicity; label all other analyses exploratory
Use alpha-spending rules for interim looks
Replicate

Perform an a priori sample size calculation using
- alpha (usually 0.05), power (usually 80-90%), the minimal clinically important difference, and the expected variance or control event rate
Increase sample size, extend follow-up, or enrich the population for events
Use the most efficient outcome measure - dichotomising a continuous variable throws away power

Post-hoc power calculation is uninformative - it is a deterministic function of the p value. Report the confidence interval instead

p value, confidence interval, statistical significance
Power, sample size calculation, minimal clinically important difference
Multiple comparisons, subgroup analysis, interim analysis
Non-inferiority margin
Publication bias, p-hacking, HARKing
Sensitivity and specificity - the diagnostic-test analogue: a false positive test is the type I error of clinical medicine

A statistically significant result with a trivial absolute effect is a real effect of no clinical value - significance and importance are different questions
"No significant difference" is not "no difference". Absence of evidence is not evidence of absence
The 0.05 threshold is arbitrary and conventional; p = 0.049 and p = 0.051 carry near-identical evidential weight
The probability that a "significant" finding is actually true depends on the prior plausibility of the hypothesis. In a field testing implausible hypotheses, most significant findings are false (Ioannidis)
Non-inferiority trials invert the usual incentives: sloppiness, dropout and poor adherence all push toward the desired conclusion of non-inferiority

11 more sections, plus exam facts

Premium unlocks every note across every specialty, and the full exam fact library behind it.