Confidence intervals and tests Updated November 2026

Type I and Type II errors

A Type I error is rejecting a null hypothesis that is actually true, a false positive; its probability is the significance level alpha. A Type II error is failing to reject a null hypothesis that is actually false, a false negative; its probability is beta, and power = 1 - beta.

Syntax

P(Type I) = alpha;  P(Type II) = beta;  power = 1 - beta

Part	What it is
`alpha`	the false positive rate you accept, usually 0.05
`beta`	the false negative rate; 0.2 is common, giving 80% power

Examples

Medical test: H0 = patient is healthy
Type I: healthy patient told they are sick
Type II: sick patient told they are healthy

Result: false positive vs false negative

A/B test at alpha = 0.05 with no real difference between A and B

Result: 5% of such tests will wrongly declare a winner (Type I)

Test with power 0.8 when a real effect exists

Result: beta = 0.2, so 20% of such tests miss the effect (Type II)

Common mistakes

Lowering alpha to 0.01 without adding data; it cuts false positives but raises beta, so more real effects are missed.
Treating a non-significant result as proof of no effect; with low power a Type II error is likely.
Running many tests at alpha = 0.05 and reporting the hits; with 20 tests the chance of at least one Type I error is 1 - 0.95^20 = 64%.

Practice Type I and Type II errors

These drills check your answer on live data, free, no account needed.

Critical t value for an intervalYou will build a 90% confidence interval for a mean from a sample of 5 observations, with the population standard deviation unknown. What critical value t* should you use? Round to 3 decimal places.

Questions

Which error is worse?

It depends on the cost. Convicting an innocent person (Type I) is treated as worse in court; missing a disease (Type II) is worse in screening. Set alpha and the required power to reflect those costs.

How do I reduce both errors at once?

Increase the sample size or reduce measurement noise. Everything else trades one error against the other.

Is a Type I error the same as a false positive?

Yes. Type I = false positive, Type II = false negative.

Statistical powerStatistical power is the probability a test detects a real effect, 1 - beta. What drives it, the 80% convention and sample size formulas with examples.
p-valueA p-value is the probability of a result at least as extreme as the observed one if the null is true. How to compute it from z and t, and common misreadings.
Hypothesis testing stepsThe five steps of a hypothesis test: state H0 and H1, choose alpha, compute the test statistic, find the p-value or critical value, decide. Worked example.
Bayes' theoremBayes theorem updates a probability with new evidence. The formula, a medical screening example with exact numbers, the natural frequency shortcut and errors.