Survivorship bias
When you only see the survivors, you draw the wrong lesson.
- When you only see the survivors, you draw the wrong lesson.
- Operating: The 'In Search of Excellence' (1982) studies of great companies were the management bestsellers of their decade.
- Investing: Backtested investment strategies look great because the universe of stocks tested usually excludes companies that went bankrupt and disappeared.
- Everyday life: Every retired person who 'lived to 95 by drinking a glass of red wine every day' is the survivor of a very large set of people who tried similar things and didn't live to 95.
The classic story. World War II, US Air Force studying which parts of returning bombers got hit most often. The natural conclusion: armour those parts. The statistician Abraham Wald said no, do the opposite. Armour the parts that don't have hits. Because the planes that got hit there didn't come back to be measured.
That's survivorship bias. You're studying the winners (or in Wald's case the survivors) and missing the systematic information sitting in the losers (or the casualties).
Once you see it, you can't unsee it:
Startup advice. The library of "how I built my company" books is written by founders whose companies succeeded. There's almost no library of "I did almost exactly the same things, and it didn't work." Without the second library, the lessons from the first are basically useless. Surviving the early years has a huge component of luck that the survivors rarely credit.
Mutual fund performance. Funds that perform badly get closed. The historical performance of "the fund family" shows the survivors. Look at the 5-year returns of all funds that existed 5 years ago (including the ones that died), and the picture gets a lot less impressive.
"Successful people wake up at 5am." Maybe. Or maybe a lot of people wake up at 5am and didn't end up running Fortune 500 companies, and we just don't read books written by them.
Building advice. "Famous old buildings were beautifully designed." Of course they were. The ugly ones got torn down. The buildings still standing are a filtered set, not a random sample.
The practical defence is to ask, every time you're about to draw a lesson from successful people or companies: "What happened to the ones who tried this and didn't make it? Would I be able to tell the difference between the successes and the failures while it was happening?"
If the honest answer is "no, I couldn't have told from the inside which would survive," the lesson is mostly noise. Be especially careful when the success story is also the storyteller. Survivors write the history, and they tend to rationalise the luck out of it.
Examples in the wild
The 'In Search of Excellence' (1982) studies of great companies were the management bestsellers of their decade. Most of the companies featured did badly in the following 10 years. The study was looking at survivors at a specific moment, not at durable greatness.
Backtested investment strategies look great because the universe of stocks tested usually excludes companies that went bankrupt and disappeared. Including the dead companies cuts most strategies' returns significantly.
Every retired person who 'lived to 95 by drinking a glass of red wine every day' is the survivor of a very large set of people who tried similar things and didn't live to 95. The sample we hear from is filtered to the longest-living.
Survivorship bias is one of the mental models we apply through real cases inside the Pareto MBA — a part-time program for professionals who want to think clearly about business.