Thirty years ago, GE became the brightest star in the firmament of statistical ideas in business when it adopted Six Sigma methods of quality improvement. Those methods had been introduced by Motorola, but Jack Welch’s embrace of the same methods at GE, a diverse manufacturing powerhouse, helped bring stardom to industrial statisticians.
Last week, GE’s glow nearly disappeared as the company cut its dividend to one penny, and word spread that it faced a criminal probe of its accounting practices. GE’s current problems lie in the arena of financial management, not statistics, but its fall does make one think of the phenomenon of “regression to the mean.” This statistical concept is like an iceberg – interesting and modest on the surface but with a breadth and depth that have great power.
Regression to the Mean (and Why it Matters)
In any process that has a component of chance variation, standout performance in one period is likely to be the product partly of luck, and so will probably be followed by a reversion to more average performance later as the luck evens out.
Consider baseball batting, and the phenomenon of “Rookie of the Year,” the term applied to the first-year player who does better than all other rookie batters. The overall probability that a batter will get a hit in a plate appearance in U.S. major league baseball is about 0.25; there is a considerable element of random chance (luck). When baseball selects the “Rookie of the Year,” it is, in effect, selecting partly for good luck. The “Rookie of the Year” is probably going to be a good player in the future, but is also very likely to decline in performance in his second year, compared to his banner first year. Why? While his underlying ability remains the same, the good luck that helped propel him to “Rookie of the Year” is unlikely to return in the same force that it did during his banner year.
This phenomenon was given its name by Sir Francis Galton 106 years ago, when he described how tall parents usually had children who were closer to average height than themselves (owing to the random elements that contribute to determining a person’s height).
Why Regression to the Mean is Important in Business
Understanding this phenomenon can help you avoid two common errors in business:
- Over-estimating the future potential of an asset or person that has recently performed very well
- Under-estimating the future potential of an asset or person that has recently performed very poorly
In both cases, the recent standout performance, either good or bad, likely has a component of luck in it that will revert to average in future periods.
This lesson is most salient when considering the extremes – the very top (or bottom) performer – in a crowded field. Selecting the extreme value in a large sample practically guarantees that random chance is a significant contributor to the performance. Making a big deal over the top sales-person, in contrast to the #2 or #5, could lead to unwarranted conclusions about pecking order. Selecting the top-performing mutual fund is bound to lead to disappointment in the succeeding period.