Data Distributions and Statistics

The data set and its two centers

Write a data set as a list of $n$ numbers, $x_1, x_2, \ldots, x_n$. Two summaries claim to be its center.

The mean pools the values and shares the total out equally. Using the sigma notation from the sequences chapter, and writing it $\bar{x}$ (read “x bar”),

The median sorts the values and takes the middle one. With an odd count the middle value is a single data point. With an even count no single value is in the middle, so the median is the average of the two middle values. The mean does arithmetic on the sizes of the values; the median only cares about their order. That difference is the whole story of this lesson.

Compare two classes whose test scores both have a mean of $50$ and a median of $50$:

Every measure of center agrees on these two sets, and yet nobody would call them the same class. A center alone is not a summary of a data set. You also need its spread, and later its shape.

The mean balances the data

The deviation of a value is its signed distance from the mean, $x_i - \bar{x}$: positive above the mean, negative below. The mean is the one place where those pulls cancel exactly.

This holds for every data set that has ever existed, which makes it both a beautiful fact and a useless measurement. If you tried to summarize spread by averaging the deviations, you would get $0$ every single time, for tightly packed data and wildly scattered data alike. Any honest measure of spread has to defeat that cancellation first. Hold on to that; it is what forces the definition of variance later.

The mean is the least-squares center

Saying the mean “balances” the data is a picture. Here is a sharper claim: the mean is the single number that sits closest to the data, when “close” is measured by total squared distance. Pick any guess $c$ for the center and score it with

The guess with the lowest score wins. The remarkable thing is that $S$ is a quadratic in $c$, so you already know how to find its minimum.

Two consequences are worth naming now, because the rest of the lesson leans on them. First, the leftover constant $\sum x_i^2 - n\bar{x}^2$ equals $S(\bar{x}) = \sum (x_i - \bar{x})^2$, the minimum possible total squared distance. That number is about to become the variance. Second, because the vertex sits at $\bar{x} = \frac{1}{n}\sum x_i$, moving any one data value by an amount $d$ moves the mean by exactly $\dfrac{d}{n}$. Drag one value far enough and the mean follows it as far as you like.

The median is the least-absolute-distance center

Squaring is not the only way to measure how far your guess sits from the data. Measure it with plain absolute distance instead,

and a different champion wins. Not the mean: the median.

Be precise about that last sentence rather than glossing it. With an even count the minimizer is not unique: any number between the two middle values does equally well. The convention of averaging them simply picks the middle of a tied region. With an odd count the median is the one and only minimizer.

The two theorems together explain the entire personality of these two statistics. See them race on the tiny data set $2, 4, 9$, whose mean is $5$ and whose median is $4$:

Each center wins its own game and loses the other one. Neither is the “real” center. They optimize different things.

Robustness, or why one bad value ruins the mean

Now the payoff. Return to the five houses, in thousands of dollars:

Their total is $2000$, so the mean is $400$, while the median is the third sorted value, $260$. Four of the five houses sold for less than the mean. A statistic that overstates four sales out of five is not describing this street.

The proofs above say exactly why. The mean is the vertex of a parabola whose position is $\frac{1}{n}\sum x_i$, so it is built out of the sizes of the values. Squared distance punishes a far-off point brutally: the mean has to move toward the mansion to keep that penalty down. The median is the innermost point of a nest of intervals built out of the order of the values. Sliding the largest value further out to sea changes the outermost gap, but it does not change which value is in the middle. So the innermost interval, and therefore the median, does not move at all.

A value like $990$, sitting far away from the bulk of the data, is an outlier. A statistic that an outlier can drag around is called sensitive; one it cannot is called resistant or robust. The mean is sensitive, the median is resistant, and now you know it is not a coincidence but a consequence of what each one minimizes. This is why house prices and household incomes are reported as medians in the news, and it is worth being suspicious when someone reports a mean instead.

Shape, and what skew does to the centers

Sort a data set into bins of equal width and draw a bar for the count in each bin, and you have a histogram. Unlike a bar graph of categories, the bars sit on a number line and their order means something, so the picture shows you the shape of the distribution.

Four shapes are worth naming. A symmetric distribution looks the same reflected about its center. A right-skewed (or positively skewed) one has a long tail of large values stretching to the right, as incomes and house prices do. A left-skewed one has a long tail to the left, as exam scores do when most of the class does well and a few people bomb it. A bimodal distribution has two separate peaks, which is almost always a sign that two different groups have been mixed together.

For a symmetric data set the mean and the median are equal, and that one is a theorem, not a slogan. Pair each value with its mirror image about the center of symmetry, note that each pair averages to that center, and both statistics land there.

Skew is a different matter, and here you should be careful, because textbooks routinely overclaim. The useful rule of thumb is that a right-skewed distribution usually has its mean above its median, and a left-skewed one usually has its mean below. The reason is that the mean feels the size of every value in the long tail while the median only counts how many values are out there. That rule is reliable enough to be worth knowing, and it is the reason a gap between the mean and the median is a good skew detector. It is not a theorem: mathematicians have constructed data sets with a long right tail whose mean sits below the median. Treat “the mean is pulled toward the tail” as a strong hint about the data, and then go look at the histogram rather than trusting the slogan.

The bimodal case deserves a warning of its own. If a distribution has two clear peaks, both the mean and the median can land in the valley between them, at a value that describes almost none of the data. Think of commute times in a town where half the people walk to work in $10$ minutes and half drive for $50$: a center of $30$ minutes describes nobody’s morning. When you see two peaks, the right move is not to pick a better center. It is to report the two groups separately.

Spread, part one: the range is fragile, the IQR is not

The crudest measure of spread is the range, $\text{max} - \text{min}$. Its problem is now easy to state precisely: it is computed from the two most extreme values in the data. Those are exactly the two values most likely to be outliers. The range is not merely sensitive; it is built entirely out of the data’s least trustworthy points.

The fix is to throw away the ends and measure the middle. Sort the data and cut it in half at the median. The first quartile $Q_1$ is the median of the lower half, and the third quartile $Q_3$ is the median of the upper half. (When the count is odd, leave the middle value out of both halves; this lesson uses that convention throughout.) Roughly a quarter of the data lies below $Q_1$, and roughly a quarter lies above $Q_3$, so the box between them holds the middle half of the data. Its width is the interquartile range:

The IQR is resistant for the same reason the median is. Push the largest value out to infinity and it is still just “some value in the top half”; it changes no quartile, so the IQR does not move.

That last claim needs one honest qualification, and it is the kind textbooks skip. It holds only when there is enough data for the extremes to sit outside the quartile calculation. With three values, $Q_3$ is the largest value, and with four values it is the average of the top two. So in those cases the outlier is part of $Q_3$, and dragging it does move the IQR. The set $1, 2, 3, 4$ has $Q_3 = 3.5$ and an IQR of $2$, but change the $4$ to a $400$ and $Q_3$ becomes $201.5$. The same caveat applies to the median when there are only two values. For any data set large enough to be worth summarizing, the extremes fall outside the box and the resistance is real, which is the case worth remembering.

Taken together, the five-number summary is

and a box plot draws it: a box from $Q_1$ to $Q_3$ with a line at the median, and whiskers reaching out to the extremes.

That leaves the question of what counts as an outlier, and here honesty matters. There is no law of nature that defines one. The standard convention, due to John Tukey, is to call a value an outlier when it falls more than $1.5 \times \text{IQR}$ beyond the nearer end of the box:

Those two cutoffs are called the fences. The number $1.5$ was chosen because it flags very few points in well-behaved data and plenty of points in badly behaved data. Nothing forces it, and some analysts use $3$ to mark the truly extreme cases. When a data set has outliers by this rule, a box plot draws the whiskers only out to the most extreme values inside the fences. The plot also draws the outliers as separate dots, so the picture shows you the exception instead of hiding it inside a long whisker.

Spread, part two: variance and standard deviation

The IQR is the spread that belongs to the median: both are built from order, and both are resistant. Now build the spread that belongs to the mean, from squared distance.

You cannot simply average the deviations, because the first proof in this lesson showed that they always add to zero. Two repairs are available. You can take absolute values, which gives an honest measure called the mean absolute deviation. (Take those absolute deviations from the median instead, and you get the spread that genuinely belongs to it, by the second theorem above.) Or you can square the deviations, which kills the signs just as effectively and punishes a far-off value much harder than a near one. A deviation of $10$ contributes $100$ times what a deviation of $1$ contributes. Most importantly, squaring gives the very quantity the mean already minimizes. Squaring keeps you inside the world where the mean is the champion, and it keeps the algebra smooth, because $x^2$ has no corner in it.

Average the squared deviations and you get the variance. If your list is the entire group you care about, the group is called a population, its mean is written $\mu$, and

The variance is in squared units, which is nonsense to report (squared minutes, squared dollars), so take the square root to come home to the original units. That is the standard deviation $\sigma$, and it is the honest headline number: roughly, how far a typical value sits from the mean.

The divisor: why $n$ sometimes becomes $n - 1$

Here is the question every textbook fudges, so let us do it properly. Suppose your $n$ numbers are not the whole group but a sample drawn from a much larger population whose true mean $\mu$ you do not know. You want the population’s spread, but you cannot measure deviations from $\mu$, because you do not have $\mu$. All you can do is measure deviations from the sample’s own mean $\bar{x}$.

That substitution is not innocent, and the least-squares theorem tells you exactly how it fails. That theorem says $\sum (x_i - c)^2$ is smallest at $c = \bar{x}$, so plugging the true mean $\mu$ into that sum can only give something at least as big:

with equality in exactly one case, the coincidence $\bar{x} = \mu$, where the sample mean happens to land precisely on the population mean.

Read that inequality carefully, because it is the whole point. The spread you want is the one measured from the true center, on the right. The spread you can compute is the one on the left, and it can never come out too big. It comes out too small in every sample except that one lucky coincidence, and it does so not by bad luck but by construction. That is because $\bar{x}$ is the value that makes that sum as small as it can possibly be. The sample measures its deviations from a center it fitted to itself, so it looks a little tighter than the population it came from. An estimate that misses in the same direction nearly every time, rather than erring evenly on both sides, is called biased, and this one is biased downward.

The repair is to divide by something smaller than $n$, which inflates the answer. The sample variance divides by $n - 1$:

Now the honest part. This argument proves that dividing by $n$ understates the spread, and it makes the $n - 1$ correction reasonable. But it does not prove that $n - 1$ is the exactly right amount to correct by rather than, say, $n - 0.5$ or $n - 2$. Showing that requires the expected value of a random variable, which is machinery you do not have yet. Take it as a debt this lesson leaves open, and take the rule with it:

• Use $\dfrac{1}{N}$ when your numbers are the whole population you care about (the heights of all $12$ people on this team).
• Use $\dfrac{1}{n-1}$ when your numbers are a sample used to estimate a larger population (the heights of $12$ people chosen from a city).

The divisor is a statement about what you are claiming, not about the arithmetic. Any question that asks for “the variance” without telling you which one is ambiguous and has two right answers, so expect every question here to say which one it wants.

One more property to file away: the standard deviation inherits the mean’s fragility. Squaring a large deviation makes it enormous, so a single outlier can dominate the whole sum. In the commute data, the $70$-minute value alone contributes $(70 - 28)^2 = 1764$ of the total $2300$, which is more than three quarters of it. The IQR ignores that value completely. So the pairing is clean: report the mean with the standard deviation, and the median with the IQR, and never mix a resistant center with a sensitive spread.

Choosing a summary

Every statistic in this lesson belongs to one of two families, and the families do not mix.

The mean is not a worse statistic than the median. It is the right one whenever the total is what you actually care about, since $n\bar{x}$ is the total. To work out a per-person share of a bill, the mean is the only correct answer and the median is useless. The failure mode is not using the mean; it is using the mean to describe a typical value in data that is skewed. Look at the shape first, then choose.

Common mistakes