Statisticsdrillp-value
The p-value is the probability, assuming the null hypothesis is true, of getting a test statistic at least as extreme as the one observed. A small p-value (below 0.05 by convention) means the data would be unusual under the null, so the null is rejected.
Syntax
two-sided p = 2 * P(Z > |z|)| Part | What it is |
|---|---|
z or t | the observed test statistic |
two-sided | counts extreme results in both directions; one-sided counts one direction only |
Examples
z = 2.0, two-sided
2 * (1 - 0.9772)Result: p = 0.0455
Excel: =2 * (1 - NORM.S.DIST(2, TRUE))
z = 2.0, one-sided (upper)
1 - 0.9772Result: p = 0.0228
t = 2.5 with 24 degrees of freedom, two-sided
=T.DIST.2T(2.5, 24)Result: p = 0.0197
Chi-square = 4 with 1 degree of freedom
=CHISQ.DIST.RT(4, 1)Result: p = 0.0455
Common mistakes
- Reading p = 0.03 as a 3% chance that the null hypothesis is true; it is the chance of the data given the null, not the chance of the null.
- Calling p = 0.20 proof of no effect; a large p means the data is inconclusive, not that the effect is zero.
- Reporting a one-sided p when the hypothesis was two-sided; that halves the p-value and overstates the evidence.
Practice p-value
These drills check your answer on live data, free, no account needed.
- Two-sided t test: p-value and decisionTest H0: μ = 68 against H1: μ ≠ 68 for the average order value (dollars). A random sample of 11 gives x̄ = 69.33 and s = 29.5. Find the p-value of the one-sample t test, then decide at α = 0.05. Round to 4 decimal places.
- One-sided t test: p-value and decisionA process engineer believes a new layout cuts the mean number of units packed per hour from the old 45. Test H0: μ = 45 against H1: μ < 45. A random sample of 10 hours gives x̄ = 44.09 and s = 1.7. Find the p-value, then decide at α = 0.05. Round to 4 decimal places.
- A/B test: p-value and decisionAn A/B test of a signup form: control 177 signups out of 1,500 visitors, variant 174 out of 1,200. Using a two-sided two-proportion z test with a pooled standard error, what is the p-value? Then decide at α = 0.05. Round to 4 decimal places.
Questions
What does p < 0.05 mean?
If the null hypothesis were true, a result this extreme would occur less than 5% of the time. By the usual convention that is called statistically significant.
Is a smaller p-value a bigger effect?
No. p depends on both effect size and sample size. A tiny effect in a huge sample can have p < 0.001; report the effect size and a confidence interval alongside p.
How do I get a p-value in Excel?
From a statistic: T.DIST.2T(t, df), 2 * (1 - NORM.S.DIST(z, TRUE)) or CHISQ.DIST.RT(x, df). From raw data: T.TEST, CHISQ.TEST or Z.TEST.
Related
- Hypothesis testing stepsThe five steps of a hypothesis test: state H0 and H1, choose alpha, compute the test statistic, find the p-value or critical value, decide. Worked example.
- Type I and Type II errorsA Type I error rejects a true null (false positive, probability alpha); a Type II error misses a false null (false negative, probability beta). Examples.
- One-sample t-testHow to run a one-sample t-test: compute t = (x-bar - mu0) / (s / sqrt(n)), find the p-value with n - 1 degrees of freedom and decide. Example and Excel.
- A/B test significanceHow to tell whether an A/B test result is statistically significant: compute relative lift, run the two-proportion z-test, read the p-value and avoid peeking.