The Flaw of Averages: Why We Underestimate Risk in the Face of Uncertainty

by Sam L. Savage

A must-read for anyone who makes business decisions that have a major financial impact. As the recent collapse on Wall Street shows, we are often ill-equipped to deal with uncertainty and risk. Yet every day we base our personal and business plans on uncertainties, whether they be next month’s sales, next year’s costs, or tomorrow’s stock price. In The Flaw of Averages , Sam Savage­known for his creative exposition of difficult subjects­ describes common avoidable mistakes in assessing risk in the face of uncertainty. Along the way, he shows why plans based on average assumptions are wrong, on average, in areas as diverse as healthcare, accounting, the War on Terror, and climate change. In his chapter on Sex and the Central Limit Theorem, he bravely grasps the literary third rail of gender differences. Instead of statistical jargon, Savage presents complex concepts in plain English. In addition, a tightly integrated web site contains numerous animations and simulations to further connect the seat of the reader’s intellect to the seat of their pants. The Flaw of Averages typically results when someone plugs a single number into a spreadsheet to represent an uncertain future quantity. Savage finishes the book with a discussion of the emerging field of Probability Management, which cures this problem though a new technology that can pack thousands of numbers into a single spreadsheet cell. Praise for The Flaw of Averages “Statistical uncertainties are pervasive in decisions we make every day in business, government, and our personal lives. Sam Savage’s lively and engaging book gives any interested reader the insight and the tools to deal effectively with those uncertainties. I highly recommend The Flaw of Averages .”
— William J. Perry , Former U.S. Secretary of Defense “Enterprise analysis under uncertainty has long been an academic ideal. . . . In this profound and entertaining book, Professor Savage shows how to make all this practical, practicable, and comprehensible.”
— ­Harry Markowitz , Nobel Laureate in Economics

Genres: Business, Nonfiction, Science, Mathematics, Economics, Management, Professional Development

562 pages, Kindle Edition

First published January 1, 2009

About the author

Sam L. Savage is a Consulting Professor of Management Science and Engineering at Stanford University, and a Fellow of the Judge Business School at the University of Cambridge.

Reviews

[Blakely · Rating 4 out of 5]

If you've read this book you'll find it quite ironic that I'm giving it a rating of 4. A real rating of the book would look like this:


5 xxxxxxxxx
4.5 xxxxxxxxxxx
4 xxxxxxxxxxxxx
3.5 xxxxxxxxxxxxxxx xxx
3 xxxxxxxxxxxxxxxxxxxxxxx xxxxxxxx
2.5 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
2 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
1.5 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
1 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
0.5 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
0 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
start of book end of book

Basically the first 100 or so pages of the book are wonderful! Very interesting, practical, and even amusing in places. Unfortunately about halfway through the book gets bogged down in discussing software programs to meet various predictive needs and since that is of no interest to me - and, I suspect, the average reader - this gets boring pretty quickly. The book does pick up towards the end.

I would recommend the book, but perhaps just start skimming if it becomes boring.

[Syed Ashrafulla · Rating 2 out of 5]

Sam Savage's flaw in the Law of Averages is roughly two-fold: dependence between variables and nonlinearity in the payoff. He forgets about the big one (small samples & non-Gaussianity kill the law), but that's not where I soured. I soured with his cutesy attempt to make probability accessible to everyone. I just don't like SIPs and SLURPs and whatever else is being used to just describe realizations of a random variable.

I understand, statistics is in my expertise, so I'm a bit biased. But come on, calling it a SLURP!? How about a realization? That's extremely easy to understand. The hard part of probability is not its basic terminology; it's the ludicrous laws governing the distribution of the sum (and thus any function) of two variables. It turned me off at the beginning and at the end.

His attempt to convert people from parametric distrbutions (Gaussian, chi-square, etc) to nonparametric distributions (using raw samples) is noble. In fact, in many places that turn has already been made. However, he is forgetting that nonparametric distributions are also estimators with massive variation as compared to parametric ones. If you know the data is Gaussian, just use the Gaussian PDF. You get a much better estimate. That's why a lot of statistical research uses hybrid approaches: non-parametric when necessary, parametric when possible.

The good part of the book, the 2 stars in it, are the examples of non-linear payoffs. For example, the licorice drug was a fantastic description of how to understand that distributions do not work like numbers. They don't add nice, they don't do anything nice. I would focus on these examples as the best and brightest and most illuminating parts of the book.

Savage does a salesman's job with respect to all the analytics software and spreadsheets, but I do wonder if the spreadsheet as a data device is in need of dismissal. With the accessibility of simple coding structure (Python & MatLab for example), why not just keep the data in whatever format, have a reading function, and then work in code? The only advantage spreadsheets have is some visualization aspects, but I'm worried that spreadsheets force bad behavior. Maybe that was the idea behind the differing GUIs from Shell and Merck.

Anyway, this book is a quick read if you want to understand at an intuitive level as to why using means without appropriate modeling is a bad idea. His message to use Monte Carlo simulation is completely accurate and is the key takeaway. I would just gloss over some of the silly terms and the weird Probability Management pitches.

[Fvandrog · Rating 1 out of 5]

I was tremendously disappointed with this book. I got the false impression that it would focus on statistics in all aspects of life -- instead it is on discussing business cases, which makes it rather boring. In addition, statistical principles are discussed in very lay terms, without any of the actualy interesting mathematical details.

[Mike Sanders · Rating 2 out of 5]

Savage takes a key idea, explains it well, and then falls into the same trap he warns against. The idea that averages are misleading, that scenarios and probability thinking is more useful is well explained at the start. But where Savage misses out is when he spends most of the book trying to ignore this and aim for precision in thinking regardless. While his approach is better than that of pure averages, merely adding an additional factor doesn't address the Flaw Of averages, but merely moves the average you're considering to a new one. Most of the book also focuses on what feels almost a sales job for varied excel tools, which again devalues the key message at the start. The firs part is worth a read over and over again, the rest is not worth anything more than a rough skimming

[E Aktas · Rating 5 out of 5]

Don’t just check the book’s score; check the variability of it too

This is an excellent and mostly easy to read book on common traps we fall in by guiding our decisions using a single figure, mostly the average of the variable of interest. The examples given make the flaws become vivid. Prof Savage introduces several memorable terms in English, making it more efficient to share ideas on uncertainty. The terms that contradicted the beginnings of the book are SIP and SLURP; difficult to remember. In any case, this was an informative and enjoyable read.

[InvestingByTheBooks.com · Rating 3 out of 5]

“Consider the state of a drunk, wandering around on a busy highway. His average position is the centerline, so the state of the drunk at his average position is alive, but the average state of the drunk is dead.” Turn on your computer, picture-google “The flaw of averages” and take a look at the brilliant picture that illustrates the quotation above. The insight behind this picture could have saved investors a lot of grief during financial history. Sam Savage is a consulting professor of management science and engineering at Stanford, a risk management software entrepreneur and the son of statistician Leonard Jimmie Savage who pioneered bayesian statistics in the 50’s and 60’s. With this book Savage follows in his father’s footsteps and explores how risk and uncertainties can be understood and managed.

Savage presents a strong and a weak form of “the flaw of averages” where the weak form is to use a single number to represent an uncertain outcome instead of letting a distribution of possible outcomes guide you and the strong form states that the average of expected inputs doesn’t always result in average or expected outputs. The essence of Savage’s insights is that no one should base their view of events on an average number but instead should use estimated probability distributions. When doing the analysis interrelationships, constraints and hidden options must be taken into account or else the decision made will be based on faulty assumptions. Savage points to the numerous mistakes made in a wide array of areas when it comes to truly understanding the implications of the various options presented. The solution presented is to accept uncertainty and manage it through appointing a Chief Probability Officer who can make use of the ever growing computing power and appropriate software solutions. As it happens Savage is also a supplier of such software.

This is actually a hard book to grade. On the one hand I have the deepest sympathies for the many aspects Savage discusses. They are extremely important in the financial world. On the other hand I find the book unnecessarily bantering, unstructured, a bit too self-promotional and Savages constant effort to distance himself from statistical academia frankly becomes a bit juvenile after a while. As this book is in part written as a biography of the author it becomes clear that he is brought up in the epicentre of the academic explosion of modern portfolio theory. The family’s friends included Milton Friedman, Harry Markowitz, Bill Sharpe, John Taylor etc. Savage is in part rebelling against the simplistic use of normal distributions which in my view is excellent, but that doesn’t motivate his somewhat petty need to downplay academia.

The financial industry is a continual villain when it comes to the use of simple averages when taking decisions. In asset allocation we often use the average historical return as the future expected return of an asset class and we use the average (and normally distributed) volatilities and correlations to estimate the robustness of the portfolio we are constructing. In doing this we too seldom consider if we can withstand the consequences of what fat tails imply and we are oblivious to the fact that correlations are totally different in shifting environments. Interrelations that on average are insignificant can kill you in times of trouble (take a long look at the drunk on the highway again). Further, very few understand the option imbedded in cash as capital preservation in times of trouble gives you the chance to reinvest when other assets have become really cheap and their expected return hence has become high. Letting software visualize the uncertainties by displaying the distributions of outcomes could be a great help.

The flaw of averages is a manifesto promoting understanding and management of uncertainties – we should all adhere to this. I just wish the book could have done the topic justice.

[John Petrocelli · Rating 3 out of 5]

Review: Opted not to read pages 173-243 (much of which focuses on applications in Finance and Real Finance - probably useful advice for investing, but very boring). The book has a few nice examples of how people overuse and abuse averages, and show how they can be nothing like what people actually experience (e.g., retirement funds over 20 years). However, the book becomes increasingly focused on finance and the stock market, and increasingly boring. Savage's writing style is slow and can put most readers to sleep after a couple of short chapters. I did like Chapter 18: Simpson's Paradox and Chapter 20: Taking Credit for Chance Occurrences.

Favorite Quote: “Wainer describes how billions of dollars were wasted, breaking large school districts into smaller ones because some well meaning but boneheaded group of do-gooders had noticed that high average test scores occur more often in small districts than in large ones. If the same group had looked at low average test scores instead, they would have noticed that low scores also occur more often in small districts, and the billions of dollars would have been wasted on consolidating smaller districts into larger ones instead of the other way around. Wainer refers to the basic law of diversification as the ‘Most Dangerous Equation’ because being oblivious of it has led to a series of misguided, costly, inept, foolhardy, and counterproductive decisions spanning nearly 1,000 years.” (pp.137-138).

[Andrew Eil · Rating 4 out of 5]

While a bit uneven in its focus and degree of technicality, Sam Savage delivers an eminently readable and accessible volume breaking down probability management into its core principles and component parts. He is refreshingly straightforward and conversational in tone, not turgid or abstruse. He also focuses on the applied science of probability, in particular risk management for enterprises. He dispenses with 'steam era' statistical methods and abjures 'flaws of averages', i.e. frequent misapprehension of numbers from the probabilistically illiterate. He makes a strong case for "mindles" (mnemonics for understanding probabilities), probability distributions over averages, visualization of probability distributions, distribution strings, probability management software, and simulations. He convincingly argues that well-designed simulation mapping based upon simulation libraries with well-defined interrelationships of variables can dramatically simplify and improve risk management and business decision-making.

[Adam Kecskes · Rating 2 out of 5]

I had this book recommended to me by a boss whose business acumen I admired. Unfortunately, I didn't admire this book at quite the same level.

It's been awhile since I read it, so I can't offer specifics, but what I do recall was that there was nothing particularly unique about the book and its offerings. The stories were bland and stereotypical of the science and math that might find in any other math/statistics oriented book. This book is essentially just another in a long line of "pop-math" books to fill the shelves of bookstores alongside the pop-psychology, pop-economics, and other pop-* books that people find so... popular.

It wasn't so bad that it deserved only one star; there's just nothing new, inspiring, or even useful to takeaway. You'll find better, more entertaining, and more engaging content for free on YouTube.

[Nick · Rating 2 out of 5]

The first half of the book was insightful and offered good perspectives on the analysis of data and how to interpret results from different angles. The basic lesson of the book was that we shouldn't rely solely on averages and should instead assess the upper and lower ranges of results as well as understanding the impacts of making decisions based on trends and probabilities.
I feel like the core theory of the book could have probably been explained in a long form blog post. It's not a particularly difficult concept and I feel like the author just padded it out to 366 pages with miscellaneous stories and references to other authors and books that he found interesting or that could help prove his point.

[Henry · Rating 5 out of 5]

- Averages are hysterically bad metric of measurement (a statisticians butt is in the freazer and head is in the oven - on average he is comfortable) because the single number doesn't capture the reality of the state: in fact, the reality would be anything but average (either higher or lower)

- In addition, sticking with this one magic number thinking is dangerous in decision making. When a person rely on multiple single number metrics that are dependent of each other, one's model would be flawed since it doesn't take account of the dependent (for instance, Red Lobster's endless lobster loss a lot of money because 1) they forget that customer orders more in buffett and 2) when they order more, lobster price goes up as store has to buy more, driving price up)

[Johan Dahlbäck · Rating 4 out of 5]

The average value disease is alive and well in management. I knew it before and I see it even clearlier after reading this book. Getting from taking mean values for truth to using distribution is a long way to go and Sam Savage gives you a lot of help.

Anyone who makes ANY business decision should know some statistics and could make good use of this book. It helps to know math, but you don't have to to get the points in the book.

I liked the chapters about options and the "mindles" most. The chapters about Probability Management history I could have done without.

[Jason Gaby · Rating 4 out of 5]

4.5/5 stars. As an economist and lover of all things probability and statistics, I resonated with much of what Savage had to say in this book. I enjoyed his simple laymen discussion on many of the flawed ways we use humans, and just how poor our intuition can be about distributions. I found the first half of the book particularly salient and enjoyable. The last half focuses more on applications which some may or may not find interesting. I knock a half star off for that.

I would highly recommend this book to anybody.

[Paloma · Rating 2 out of 5]

This is the sort of concept that would make for a great article and just gets dragged on far too long to be a book. The main idea is extremely important for anyone in the position to make business decisions and how the mean, probably our favorite statistic, can be extremely misleading and cause us to make poor decisions. Still, I found myself dragging my feet through many of the sections of this book until finally giving up at the halfway point.

[Sasha Kaufmann · Rating 3 out of 5]

A book on assessing risk in the face of uncertainty, focussing as the title suggests on how not to assess it. The author is quirky and he makes up for the dryness of the subject by packing the chapters with examples and analogies; he's funny (occasionally). I didn't find anything drastically new but it was a good refresher.

[Preston · Rating 3 out of 5]

I was recommended this book by one of my Operations Research Professor's and found it to be pretty entertaining while educational. I think for someone who isn't familiar with Operations Research or Statistics would find this book much more insightful than me, regardless I still enjoyed it.

[Evan Buehler · Rating 4 out of 5]

First half of the book is fantastic. Second half of the book gets too technical and detailed with the software programs. Overall, a humorous book which gives a unique statistics perspective that isn't taught in stats 101.

[Peter · Rating 5 out of 5]

Insightful!

[F_alsaad · Rating 1 out of 5]

What a disappointment!!!!

[Fernando Cuenca · Rating 2 out of 5]

The book succeeds in presenting the pitfalls of using just averages to make decisions, but I'd argue that it could have been done in less than its 300+ pages.

[Johanna Rothman · Rating 5 out of 5]

I did not expect to enjoy a book about averages. I loved it! Not only did I learn ways to discuss why averages don't work, I have other alternatives.

A great book.

[Riley · Rating 1 out of 5]

sam savage discovers the concept of biased estimators and blames statistics for people applying averages where they don’t apply

[Mattia Battiston · Rating 5 out of 5]

Fantastic Read! Whenever faced with uncertainties use distributions and simulations, not averages.

I work in IT - software development - and I've always had to deal with plans and estimates. This book explains with extreme clarity why the traditional way of using averages for such reasons is wrong in so many ways!

I already knew the saying that you're not meant to use averages as they are wrong half the time, but this book still opened my eyes on the real reasons behind it.
Some of my favourite quotes:
- "when we use single numbers to estimate uncertain future outcomes we are not just usually wrong, but we are consistently wrong"
- "Plans based on average assumptions are wrong on average"
- "when managers ask for a forecast, they are really asking for a number, which involves the Flaw of Averages"
- "risk is in the eye of the beholder"
- "it is essential to replace numbers with distributions to cure the flaw of averages"
- "weak form of the flaw of averages: a single number doesn’t give the whole picture"
- "the strong form of the flaw of averages states that average or expected inputs don’t always result in average or expected outputs"
- "information has no value at all unless it has the potential to change a decision"
- "the simulation of the sum is not the sum of the simulation"

By the way the book is well written. The author is funny and has an easy-to-read style - he manages to make hard topics like finance and economics easy to understand even for me that I know very little about them.

The bottom line for me is: whenever faced with uncertainties (which is the norm in software development) use distributions and simulations, not averages.

[Brad Felix · Rating 5 out of 5]

I'd recommend this book to anyone who works for a corporation that uses numbers.

In my professional experience, I've found that managers often ask for your best prediction on some uncertain outcome and when you try to explain all the underlying variables and distributions of those variables that could influence the outcome - they typically reply with an incredulous "Give me a number!" Without a sophisticated way to model or simulate the results, most analysts are left to finding the average of historical data and giving that as the best prediction of future results.

This is the problem Sam Savage addresses, not only with this book but with SIPMath - an open-source, Excel plug-in that allows anyone to perform instantaneous Monte Carlo simulation on any computer. I believe him when he says that simulation becoming a free, commodity-like tool will transform how analysts and businesses make decisions in the same way that the invention of the spreadsheet changed how people made decisions prior to that time. The concepts in this book are probably some of the most important things I've ever read - and, honestly, I'm not completely sure how to apply them all to the way I analyze data and create models. But, I can guarantee you that I will never use the average of historical data. And, I can guarantee that my decisions will come from a place of deeper understanding how uncertainties actually work and should be analyzed in real life.

[Vadim · Rating 2 out of 5]

Эта без сомнения умная книга написана столь же хорошо, сколько и утомительно. Глава за главой автор уговаривает нас не делать действительно распространенную ошибку: полагаться в наших планах на средние значения. Хотя тот, кто идет по середине дороги с двусторонним движением, не пострадает, пьяница, который идет по середине лишь "в среднем", будет "в среднем" же мертв. В общем, нужно думать не про средние значения (в среднем у каждого человека одно яичко и одна грудь, но это бессмыслица), а на диапазоны возможных значений. Недостаток книги в том, что он продолжает уговаривать, когда уже любой читатель давно с ним согласен, и ищет выхода. Об этом в книге меньше, и все же его вывод ясен: вместо сложных вычислений можно воспользоваться одной из уже немалого количества демократичных в использовании компьютерных стимуляторов неопределенных процессов. Такие симуляторы должны сделать операции с диапазонами (распределениями возможных исходов) столь же доступными, как Excel сделал работу с числами. Примеры, которые приводятся на сайте книге, не оставляют сомнений в том, что автор знает, о чем говорит.

[R · Rating 3 out of 5]

The book is a great introduction to some simple concepts of flaw of averages. The first half of the book does a great job in explaining the flaws of averages such as Jensen's inequality and some application of these flaws in real life. However the author keeps on repeating the same concept over and over again throughout the book in different domains such as finance, operations and trading. Most of the concepts covered in the book would be covered theoretically in the undergraduate statistics course or more practically in the MBA's quantitative analysis course. If you have taken any of these courses, this book may just be a repetition of the course, otherwise it may serve as a good introduction to flaws in managerial decision making. Nevertheless the book could have been half its size and could have been written in a better manner than just a collection of a few hundred blog entries over the course of time.

[Abe Farag · Rating 5 out of 5]

loved this book. For my it was really eye opening to see so many ways that well intention-ed management direction could result in such poor operational outcomes. So many project management decisions and directions when designing new hardware, software, or physical products are related to asking for and executing strategy related to guesses about the future. The Flaw of averages spells out some common pitfalls in those guesses.
In my experience managing product design projects in for technology and consumer products this book is quite useful & way novel information.

But I can see that if you have this material already in a stats class or in other classes then this book could be longer than needed. Also a downside from a education take away is that it is written in a style of a group of cool things which rather than a style might just much easier than writing a book that has some arch & some cohesive outline of what it is teaching.

- Abe Farag

[Whitfield · Rating 5 out of 5]

Currently, a tectonic shift is occurring in the way statistics, probability, and simulations are being used for making decisions in the face of uncertainty. With this book, Professor Savage has captured a few of the core elements of this shift, has provided some history, and has pointed towards some future directions. Given how dry these topics can be, he has done well to make the book approachable and entertaining.

While the book meanders a bit, its principles are powerful. Interactive simulations that don't need to assume independence and that capture interrelationships will make uncertainty more apparent and intuitive. Chances are good that within 5 years, your company will be using some of these methods for making decisions in engineering, investment, or management (if they are not already).

[Tom · Rating 3 out of 5]

I usually called averages 'the curse of the average' because what does average really tell you without knowing the distribution shape? How often do people add averages (oops, unless the sample size is coincidentally the same).

The author advocates 'probability management' and creating a model that will run random tests using selected probability distribution functions.

The book's points are very good. Some one familiar with the 'theory of constraints' (like in The Goal, or It's Not Luck, or Critical Chain) would be familiar with most of the points. The writing style is a bit chatty, but maybe that is important in a book related to statistics. I found some of the font usage and pooh-poohing the vocabulary distracting, but if it helps with making the topic more accessible, then those too are small inconveniences

[Gary · Rating 5 out of 5]

This is a great book - if you want to understand why humans are almost always wrong when they make estimates. Dr. Savage looks at the way statistics has been and is being taught and points out how the advent and now common use of the digital computer makes much of the current pedagogy obsolete. Likewise, he offers real, applied examples of how variance and risk assessment can and are being used.

I enjoyed this book enough that I've given copies to a number of colleagues, clients, and business associates. It is unfortunate more of the 'quants' didn't use this type of thinking earlier in the past decade. If they had done so we might not have suffered the economic meltdown - or not suffered it to the same degree. Well done, sir!