Outliers
An outlier is a value far from the rest of the data. The common test is the IQR rule: anything below Q1 - 1.5 * IQR or above Q3 + 1.5 * IQR is an outlier. For roughly normal data, a z-score beyond 3 (or beyond 2 for a stricter screen) is another common flag.
Syntax
lower fence = Q1 - 1.5 * IQR; upper fence = Q3 + 1.5 * IQR| Part | What it is |
|---|---|
Q1, Q3 | the first and third quartiles |
IQR | Q3 - Q1 |
Examples
Data: 2, 4, 5, 6, 7, 8, 30
Q1 = 4, Q3 = 8, IQR = 4
fences: 4 - 6 = -2 and 8 + 6 = 14Result: 30 is an outlier (above 14)
z-score rule: mean = 70, sd = 10, value = 105
z = (105 - 70) / 10Result: z = 3.5, flagged as an outlier (beyond 3)
Excel: =ABS(STANDARDIZE(x, mean, sd)) > 3
Mean with and without the outlier for 2, 4, 5, 6, 7, 8, 30
with: 62 / 7 = 8.86; without: 32 / 6Result: 5.33, so one value moved the mean by more than 3 points
Common mistakes
- Deleting outliers just because they are extreme; a true measurement that is unusual is still data, only remove values you can show are errors.
- Using the z-score rule on heavily skewed data; the outliers inflate the sd and hide themselves, so prefer the IQR rule there.
- Forgetting that removing an outlier changes the mean, sd, quartiles and fences; recompute everything after a removal.
Practice Outliers
These drills check your answer on live data, free, no account needed.
- Range of a data setThe overnight low temperatures (°C) in a lake district village over 8 nights were: -12.5, 11.5, 4.5, -0.5, -3.5, -2, -8, 0.5. What is the range? Round to 1 decimal place.
- z-score of one valueLast week a call center agent handled 33 calls in a shift. Across the group, the mean is 70 with a standard deviation of 11.3. What is the z-score of this value? Round to 2 decimal places.
Questions
Should I remove outliers before calculating the mean?
Only if they are errors (typos, broken sensors, impossible values). Otherwise report the median or a trimmed mean alongside the mean and say how outliers were handled.
Why 1.5 times the IQR?
It is a convention from Tukey's box plot. For normal data it flags about 0.7% of values. A 3 times IQR fence is used for extreme outliers.
How do outliers affect correlation and regression?
A single point far from the others can create or destroy a correlation and swing the slope. Plot the data and check the regression with and without the point.
Related
- Range and interquartile rangeThe range is max minus min; the interquartile range is Q3 minus Q1. Worked examples with exact numbers, Excel QUARTILE.INC and why methods differ.
- z-scoreThe z-score formula z = (x - mean) / sd, worked examples, how to read a z-score as a percentile with the normal table, and the Excel function STANDARDIZE.
- Mean, median and modeHow to calculate the mean, median and mode of a data set by hand and in Excel, when each average is the right one, and the mistakes that give a wrong answer.
- ResidualsA residual is observed y minus predicted y. How to compute residuals from a fitted line, a worked example, and residual plot patterns that signal a bad model.