What do you think?
Rate this book
Standard Deviations: Flawed Assumptions, Tortured Data, and Other Ways to Lie with Statistics
Or that Asian Americans are most susceptible to heart attacks on the fourth day of the month? Or that drinking a full pot of coffee every morning will add years to your life, but one cup a day increases the risk of pancreatic cancer? All of these “facts” have been argued with a straight face by credentialed researchers and backed up with reams of data and convincing statistics.
As Nobel Prize–winning economist Ronald Coase once cynically observed, “If you torture data long enough, it will confess.” Lying with statistics is a time-honored con. In Standard Deviations, economics professor Gary Smith walks us through the various tricks and traps that people use to back up their own crackpot theories. Sometimes, the unscrupulous deliberately try to mislead us. Other times, the well-intentioned are blissfully unaware of the mischief they are committing. Today, data is so plentiful that researchers spend precious little time distinguishing between good, meaningful indicators and total rubbish. Not only do others use data to fool us, we fool ourselves.
With the breakout success of Nate Silver’s The Signal and the Noise, the once humdrum subject of statistics has never been hotter. Drawing on breakthrough research in behavioral economics by luminaries like Daniel Kahneman and Dan Ariely and taking to task some of the conclusions of Freakonomics author Steven D. Levitt, Standard Deviations demystifies the science behind statistics and makes it easy to spot the fraud all around.
As Nobel Prize–winning economist Ronald Coase once cynically observed, “If you torture data long enough, it will confess.” Lying with statistics is a time-honored con. In Standard Deviations, economics professor Gary Smith walks us through the various tricks and traps that people use to back up their own crackpot theories. Sometimes, the unscrupulous deliberately try to mislead us. Other times, the well-intentioned are blissfully unaware of the mischief they are committing. Today, data is so plentiful that researchers spend precious little time distinguishing between good, meaningful indicators and total rubbish. Not only do others use data to fool us, we fool ourselves.
With the breakout success of Nate Silver’s The Signal and the Noise, the once humdrum subject of statistics has never been hotter. Drawing on breakthrough research in behavioral economics by luminaries like Daniel Kahneman and Dan Ariely and taking to task some of the conclusions of Freakonomics author Steven D. Levitt, Standard Deviations demystifies the science behind statistics and makes it easy to spot the fraud all around.
304 pages, Hardcover
First published January 1, 2014
327 people are currently reading
1654 people want to read
Ratings & Reviews
Friends & Following
Create a free account to discover what your friends think of this book!
Community Reviews
Displaying 1 - 30 of 107 reviews
May 18, 2015
If you're short on time and want the most value from this book, start by reading chapter nineteen, which provides an excellent summation of the books driving themes, in just eight pages.
If you've been sold on the book after this, the one-page summary notes at the end of each chapter are your next stop. These "Don't be Fooled" sections give you the executive summary of the main ideas of the chapter, and include useful advice for use in your own analyses.
If the theses still have your interest, perhaps then read the chapters whole. The book overall is decent. Nevermind the unfortunate typos (at least in my printing). It appears the publisher's workflow suffered from a case of, spellcheck--checked!, but proofread?--meh, they must've had deadlines.
There are plenty of other great books out there covering much the same ground as this one. Some are more focused in particular topic areas and maybe give a more authoritative treatment than this one. I'd regard this book as more of a "field guide" to thinking about statistical presentations for a mass market audience. But, don't let that dissuade you from this one. I mention only to encourage those who want more, that plenty of quality material exists.
-----
That's my review, now to pick a nit:
I take issue with the treatment Dr. Smith gives the "experts" on pp. 94-96. The point he appears to be making here is, /don't just accept expert judgement, their logic may be idiotic./ After duly citing several of their works in the analysis of the "Two-Daughter Paradox", he afterward conspicuously doesn't take anyone to task by name, but rather refers to the group en masse as the "experts".
One expert he cites is Dr. Leonard Mlodinow from his 2008 book, "The Drunkard's Walk: How Randomness Rules Our Lives". (a very excellent book)
Dr. Smith takes a greatly condescending tone in addressing his fellow experts in their analysis of this problem. He advocates the judicious application of "common sense" and decries their analysis as just so much tortured/flawed logic.
I read Mlodinow's book previously, and couldn't fault his analysis then. But Dr. Smith having set fire to it, I felt I had to go back and see how I could've been so misled! My conclusion, after re-reading Mlodinow and studying some web resources as well: They're /both/ right, but each is providing the correct answer to a very subtly different question. It just so happens that language is often sufficiently vague that it's easy to slip and believe that Drs. Smith and Mlodinow are talking about the same question. In fact, they are not (which is also a theme Dr. Smith addresses very well later on in the book, all the more to grind my gears here).
The theme of the Wikipedia entry on the problem is to highlight that the problem is rather ambiguous, and the correct analysis (1/2 or 1/3) depends on the problem statement.
In deference to Dr. Smith, I think his "common sense" approach is sufficient to apprehend the probability one would encounter if faced with a question like this in the real world. It should be 1/2, and so I guess it is normally, because the way the question is posed in Mlodinow's work requires the kind of mathematical absolutism that very often trips up high school students working "always/sometimes/never" problems. The careful framing of the problem required to show the probability is actually 1/3 is not so likely if you were to meet the parent of a child in the park (to be clear: the framing of the question doesn't /change/ the probability in the answer, but we're really talking about two entirely different problems that just happen to sound like the same problem when put into words without due care). But this said, Mlodinow's analysis of the problem, as he presented it, is absolutely correct (1/3). In short, he didn't deserve the harsh treatment he and his fellow mathematical athletes got at Dr. Smith's hand.
Smith's point larger point is entirely valid: one has to be aware that one's intuition could be right or wrong when making an off-the-cuff analysis, but common sense and taking the time to do some critical thinking will serve to illuminate the problem better and raise the odds your conclusion is correct.
Smith is a Yale economist, Mlodinow is a Berkley physicist teaching probability theory now at Caltech.
(damn economists)
If you've been sold on the book after this, the one-page summary notes at the end of each chapter are your next stop. These "Don't be Fooled" sections give you the executive summary of the main ideas of the chapter, and include useful advice for use in your own analyses.
If the theses still have your interest, perhaps then read the chapters whole. The book overall is decent. Nevermind the unfortunate typos (at least in my printing). It appears the publisher's workflow suffered from a case of, spellcheck--checked!, but proofread?--meh, they must've had deadlines.
There are plenty of other great books out there covering much the same ground as this one. Some are more focused in particular topic areas and maybe give a more authoritative treatment than this one. I'd regard this book as more of a "field guide" to thinking about statistical presentations for a mass market audience. But, don't let that dissuade you from this one. I mention only to encourage those who want more, that plenty of quality material exists.
-----
That's my review, now to pick a nit:
I take issue with the treatment Dr. Smith gives the "experts" on pp. 94-96. The point he appears to be making here is, /don't just accept expert judgement, their logic may be idiotic./ After duly citing several of their works in the analysis of the "Two-Daughter Paradox", he afterward conspicuously doesn't take anyone to task by name, but rather refers to the group en masse as the "experts".
One expert he cites is Dr. Leonard Mlodinow from his 2008 book, "The Drunkard's Walk: How Randomness Rules Our Lives". (a very excellent book)
Dr. Smith takes a greatly condescending tone in addressing his fellow experts in their analysis of this problem. He advocates the judicious application of "common sense" and decries their analysis as just so much tortured/flawed logic.
I read Mlodinow's book previously, and couldn't fault his analysis then. But Dr. Smith having set fire to it, I felt I had to go back and see how I could've been so misled! My conclusion, after re-reading Mlodinow and studying some web resources as well: They're /both/ right, but each is providing the correct answer to a very subtly different question. It just so happens that language is often sufficiently vague that it's easy to slip and believe that Drs. Smith and Mlodinow are talking about the same question. In fact, they are not (which is also a theme Dr. Smith addresses very well later on in the book, all the more to grind my gears here).
The theme of the Wikipedia entry on the problem is to highlight that the problem is rather ambiguous, and the correct analysis (1/2 or 1/3) depends on the problem statement.
In deference to Dr. Smith, I think his "common sense" approach is sufficient to apprehend the probability one would encounter if faced with a question like this in the real world. It should be 1/2, and so I guess it is normally, because the way the question is posed in Mlodinow's work requires the kind of mathematical absolutism that very often trips up high school students working "always/sometimes/never" problems. The careful framing of the problem required to show the probability is actually 1/3 is not so likely if you were to meet the parent of a child in the park (to be clear: the framing of the question doesn't /change/ the probability in the answer, but we're really talking about two entirely different problems that just happen to sound like the same problem when put into words without due care). But this said, Mlodinow's analysis of the problem, as he presented it, is absolutely correct (1/3). In short, he didn't deserve the harsh treatment he and his fellow mathematical athletes got at Dr. Smith's hand.
Smith's point larger point is entirely valid: one has to be aware that one's intuition could be right or wrong when making an off-the-cuff analysis, but common sense and taking the time to do some critical thinking will serve to illuminate the problem better and raise the odds your conclusion is correct.
Smith is a Yale economist, Mlodinow is a Berkley physicist teaching probability theory now at Caltech.
(damn economists)
April 28, 2017
Pretty darn good. Almost rated it five stars if it weren't for the lack of originality: there are lots of books on errors in data analysis.
Because this is a recent book, it offers some advantages over, say, the classic How to Lie with Statistics.
Seems like it would be a perfect introductory book for anyone looking to learn more about how data analysis, data science, etc. are conducted badly. I'd add it into the data flaws canon, along with How to Lie with Statistics, The Signal and the Noise, and Fooled by Randomness. These 4 book will induce extreme LSD-level revelations in the reader.
Because this is a recent book, it offers some advantages over, say, the classic How to Lie with Statistics.
Seems like it would be a perfect introductory book for anyone looking to learn more about how data analysis, data science, etc. are conducted badly. I'd add it into the data flaws canon, along with How to Lie with Statistics, The Signal and the Noise, and Fooled by Randomness. These 4 book will induce extreme LSD-level revelations in the reader.
February 23, 2015
An enlightening and entertaining discussion of the use and, more often, misuse of statistics. In the early 60's I read How to Lie with Statistics, an absolutely fantastic expose, and this is in the same vein. I am regularly annoyed at the misuse of statistics in news articles that make huge erroneous claims based on flimsy statistics. Sometimes I think people will believe anything if some sort of data/statistic is attached. Reading this book would definitely remedy that! Examples of the false interpretation of statistics were taken from advertisements, politics, academic research, and news reports.
One kind of statistical study that has often impressed me is the analysis of large amounts of data and the discovery of patterns within that data. The author calls this technique "data mining" and shows how it can be used to "support" all sorts of ridiculous claims. I will no longer be fooled when I read that a conclusion is "based on the analysis of xxx thousands......."
Gary Smith is an entertaining writer and this book should be required reading for journalists and college students, indeed, for all of us.
One kind of statistical study that has often impressed me is the analysis of large amounts of data and the discovery of patterns within that data. The author calls this technique "data mining" and shows how it can be used to "support" all sorts of ridiculous claims. I will no longer be fooled when I read that a conclusion is "based on the analysis of xxx thousands......."
Gary Smith is an entertaining writer and this book should be required reading for journalists and college students, indeed, for all of us.
March 17, 2023
Standard Deviations reads like one of the many other books on statistical fallacies that I read. It is slightly outdated but still serves its purpose. It contains a reference to the old Cubs curse. You know, the baseball team that won the World Series in 2016 or something.
Author Gary Smith examines how people use statistics to fuel an agenda or stretch the truth. At the end of each chapter is a summary explaining how to avoid such a pitfall.
Thanks for reading my review, and see you next time.
Author Gary Smith examines how people use statistics to fuel an agenda or stretch the truth. At the end of each chapter is a summary explaining how to avoid such a pitfall.
Thanks for reading my review, and see you next time.
December 31, 2014
This insightful book explains how we should all be skeptical when presented with any information. Just because a statement is preceded by "Studies show...." or "75% of healthy people...." does not mean the information is infallible. Researchers make mistakes, data can be manipulated, and most of all patterns can be found in almost any data set from which foolish conclusions can be drawn and presented to the gullible. While this was an amusing read, there is a dark undertone due to the fact that every day we make choices based on what we assume is good data, and politicians make policies based on false information.
The fundamental fact is that to be valid, any study must start with a theory and draw conclusions ONLY about that data. And of course the methodology must be consistent and the calculations correct. One of the most depressing case studies was on a recent report from Carmen Reinhart and Kenneth Rogoff asserting that countries go into recession when total government debt reaches 90% of the total economy. This was used by conservative news outlets to stir irrational fear and had direct impacts on the national budget discussion. Unfortunately, Reinhart and Rogoff left out key data sets and made calculation errors which completely refute their conclusions. Further, there is no rational reason why that 90% should be important (as opposed to say, 75% or 92%). Because the authors wanted to get their study published (and presumably had a political agenda), they essentially falsified their report. Conservatives took it as gospel however and used it to impact our lives.
The author explains how to rationally employ skepticism to hold researchers accountable for their work. Remember that correlation does not imply causation, patterns appear in even random data sets, and that any good study starts with a theory and ends with the data, not the other way around.
The fundamental fact is that to be valid, any study must start with a theory and draw conclusions ONLY about that data. And of course the methodology must be consistent and the calculations correct. One of the most depressing case studies was on a recent report from Carmen Reinhart and Kenneth Rogoff asserting that countries go into recession when total government debt reaches 90% of the total economy. This was used by conservative news outlets to stir irrational fear and had direct impacts on the national budget discussion. Unfortunately, Reinhart and Rogoff left out key data sets and made calculation errors which completely refute their conclusions. Further, there is no rational reason why that 90% should be important (as opposed to say, 75% or 92%). Because the authors wanted to get their study published (and presumably had a political agenda), they essentially falsified their report. Conservatives took it as gospel however and used it to impact our lives.
The author explains how to rationally employ skepticism to hold researchers accountable for their work. Remember that correlation does not imply causation, patterns appear in even random data sets, and that any good study starts with a theory and ends with the data, not the other way around.
November 11, 2018
A good introduction to how data is manipulated, misused or misinterpreted. The author does a good job of using real life accounts to show how really silly policy/business decisions/individual choices can be made using plausible looking but actually false information. Probably not going to surprise anyone who has any statistics background, but a very readable introduction for those who want to understand the world of data more. This could be a good book for school students as it will prepare them to enter today's data rich world with open eyes
March 21, 2015
This book was great! Despite being about the much dreaded topic of statistics, it was highly entertaining and easy to read. Gary Smith filled his book with understandable examples and winning arguments, truely helping the reader understand how easily data is tortured and sold as fact.
February 13, 2016
Clear and mildly humorous. The best food for thought was the concept of "data without theory". It is something I've wondered about and he clarified some misconceptions I had. Also, his refutation of the common answer to a riddle I've seen many times was shocking (you know your neighbor has 2 children and one is a girl. what is the probability the other is a girl? Does the probability change if you know that the known girl is the oldest of the two children?). It would have been well to have several pages of warnings to sit down and get a pot of Celestial Seasonings Tension Tamer Tea brewed before that doozie.
July 11, 2016
My background is in Economics, so this book covers some familiar content. I can only assume that the content is pretty consumable for layman (I am biased in this sense). Nonetheless, the content was still refreshing in certain aspects, bringing in interesting stories and views on some of the statistical concepts. No worries, there are no hard equations in this book. Just some numbers that might take some time to digest. A light read that I would recommend to anyone who wants to get more knowledge on how to be careful with data.
October 26, 2014
For an academic this book was well written with great examples of how even well-intentioned researchers can be duped by numbers.
April 25, 2018
Excellent entertaining coverage of statistical logic.
March 9, 2021
Data and statistics can portray information in very misleading ways. Smith takes the reader through some of the fallacies, providing examples which are inevitably entertaining. While some intentionally misleading situations can be easily spotted, in many cases one's only defence is an awareness of possible meddling and the need to question data before accepting the conclusions drawn from the data.
Topics covered include:
- mangled graphs - those without zero on the Y-axis may be exaggerating the data variation
- hot streaks - the idea that players get hot and have a greater chance of scoring than their normal probability
- regression to the mean - successes and failures that are not representative of the capability of the player, to which future events will regress
- law of averages - believing that past events affect the probability of future events in order to bring the series back to the initial probability
- the Texas sharpshooter - testing widely until a pattern is seen, then theorizing on the pattern without further testing it
- pruned data - reporting on only that data that supports the presenter's argument
- data without theory - patterns appear in data by chance - it can be an error to assume they are significant, cancer clusters being an example
- theory without data - models where the underlying algorithm is poorly supported, such as the Club of Rome extrapolations
The book is worth reading just for the humorous examples of data mangling related by Smith.
Topics covered include:
- mangled graphs - those without zero on the Y-axis may be exaggerating the data variation
- hot streaks - the idea that players get hot and have a greater chance of scoring than their normal probability
- regression to the mean - successes and failures that are not representative of the capability of the player, to which future events will regress
- law of averages - believing that past events affect the probability of future events in order to bring the series back to the initial probability
- the Texas sharpshooter - testing widely until a pattern is seen, then theorizing on the pattern without further testing it
- pruned data - reporting on only that data that supports the presenter's argument
- data without theory - patterns appear in data by chance - it can be an error to assume they are significant, cancer clusters being an example
- theory without data - models where the underlying algorithm is poorly supported, such as the Club of Rome extrapolations
The book is worth reading just for the humorous examples of data mangling related by Smith.
April 19, 2022
I really enjoyed this book. It lays out all of the situations when the data is most likely to betray us. My favorite quote on this is "torture the data enough, and it will confess."
Smith lays out the times and situations when data is most likely to be used to obfuscate and the specific things to watch for when ever we read statistics. I have found that I've used this a lot in the past month when I hear about a study.
It's a well-written book and has some fascinating insights that will have you looking critically at any report that uses data.
Smith lays out the times and situations when data is most likely to be used to obfuscate and the specific things to watch for when ever we read statistics. I have found that I've used this a lot in the past month when I hear about a study.
It's a well-written book and has some fascinating insights that will have you looking critically at any report that uses data.
September 25, 2021
Aborda várias formas de distorcer as estatísticas, mas o livro "Como mentir com estatística" também já faz isso. Achei os últimos capítulos muito repetidos, pareceu que foram colocados só pra aumentar o livro.
Achei interessante como o autor mostra que o famoso estudo de Rogoff e Reinhart, que foi usado para justificar austeridade fiscal sob o risco de gerar recessão, teve erros de cálculo e como esses erros mostram o contrário do estudo, i.e., aumento da relação dívida/PIB é consequência da recessão.
Achei interessante como o autor mostra que o famoso estudo de Rogoff e Reinhart, que foi usado para justificar austeridade fiscal sob o risco de gerar recessão, teve erros de cálculo e como esses erros mostram o contrário do estudo, i.e., aumento da relação dívida/PIB é consequência da recessão.
June 29, 2024
A solid and very practical explanation of how statistics and "data" can be used to obfuscate and mislead the reader. "Standard Deviations" by Gary Smith goes into fantastic detail and cites multiple erroneous studies from the past and illustrates how various "tricks" can be used to mislead and misinform even the most well-intentioned and vigilant reader.
A very useful read in this "post Truth" world.
A very useful read in this "post Truth" world.
October 12, 2024
I am probably 3.5 on this. I enjoyed the book and it's a relatively quick and easy read, about the ways people misuse numbers and data.
But I was an English professor, and the writer was not well-served by his editor.
But I was an English professor, and the writer was not well-served by his editor.
November 12, 2018
I read the first two chapters. The ideas aren't original... I was hoping for a rigorous and scholarly analysis, it didn't deliver. It reads like pop-stats, I should have read the reviews first.
June 27, 2022
Assigned as a part of of a doctoral course I took this summer, this one surprised me at how engaging it was. It's written for a popular audience, so there's no academic fluff to clutter it. Overall it was really interesting and does a good job at demonstrating ways in which data can be manipulated to advance a narrative, and how easy it is to do so.
March 12, 2019
-
Bad as audiobook
Bad as audiobook
October 21, 2018
The title is really interesting but i was expecting a more rigorous treatment of the subject. While the book is very instructive on how one can use/misuse data to depict a story, it is at best a beginners guide.
May 15, 2023
After working at two corporations where the “merry marketeers” (one of the nicer nicknames for the headstrong, ambitious marketing people who wouldn’t listen to anyone else) twisted statistics into contortions that would have challenged Marvel’s Mr. Fantastic or D.C.’s Plastic Man, I resolved to go back and pay more attention to statistics. Standard Deviations: Flawed Assumptions, Tortured Data, and Other Ways to Lie with Statistics accomplishes what I had set out to do much better than I could have accomplished it. Between my challenges of marketing assumptions and my desire to teach my students (when researching virtual communities) to avoid many of these errors, I wish I’d had this book decades earlier. It would have sharpened my arguments and my presentations.
One gets a sense of where the volume is going from early on when author Gary Smith quotes Ronald Coase’s cynical comment: “If you torture the data long enough, it will confess.” (p. 19) I’d heard the quotation, but never knew the source. Then, after explaining the basis for our genetic predilection for pattern recognition (whether significant or not), the author states the problem with over relying on such an instinct: “Our inherited desire to explain what we see fuels two kinds of cognitive errors. First, we are too easily seduced by patterns and by the theories that explain them. Second, we latch onto data that support our theories and discount contradicting evidence.” (p. 21). Late in the book, Smith reiterates his warning: “Data without theory is alluring, but misleading.” (p. 307).
The beauty of the book is that it skewers the poor to abominable research practices in multiple fields: medicine, economics, sociology, parapsychology, military strategy, and even web design. His examples are derived from historical studies and modern studies, and his conclusions are clear. One’s mental alarm quickly signals what your visceral reaction to conflicting studies (particularly medical and psychological/sociological) over the years, to watch out. For example, he cites John Ionnadis of the Stanford University School of Medicine who: “looked at 45 of the most widely respected medical studies during the years 1990 through 2003 that claimed to have demonstrated effective treatments for various ailments. In only 34 cases were attempts made to replicate the original test results with larger samples. The initial results were confirmed in 20 of these 34 cases (59 percent). For seven treatments, the benefits were much smaller than initially estimated; for the other seven treatments, there were no benefits at all.” (p. 31)
In demonstrating problems, Smith cites: self-selection bias (p. 40), survivor bias (particularly in backward-looking studies—p. 56), confirmation bias (p. 68), deceptive graphing (“The ups and downs in the data can be exaggerated by putting a small range of numbers on the axis and can be dampened by putting a large range of numbers on the axis.”--p. 106), the false-positive problem (p. 136), confounding factors (p. 145), regression to the mean (p. 180), trust in the fallacious law of averages (p. 203), starting with data clusters rather than researching a theory (p. 213, otherwise known as the Texas sharpshooter approach – p. 215), be careful in discarding outliers (sometimes, you shouldn’t – p. 234), the selective reporting or “publication effect” (p. 273), use of forward or backward displacement to create a multiplicity of potentialities (p. 274), and extrapolation of past trends into future expectations (p. 288) seem to be the major pitfalls. In addition to these, Smith simply warned: “the two main pitfalls of quantitative financial analysis: a naive confidence that historical patterns are a reliable guide to the future, and a dependence on theoretical assumptions that are mathematically convenient but dangerously unrealistic.” (p. 331).
One of Smith’s most interesting examples to me (since I enjoy military history) was reading about a WWII effort to study damaged planes so that they would have a statistical basis for placing additional (albeit heavy) armor on vulnerable areas. Yet, the self-selecting fallacy in this study was that planes hit in the most vulnerable places like the cockpit and fuel lines didn’t usually make it home. Thanks to an analysis by Andrew Wald, authorities discovered: “Returning planes were more likely to have holes in the wings because these holes did little damage. Wald’s advice was exactly the opposite of the initial conclusion. Instead of reinforcing the locations with the most holes, they should reinforce the locations with no holes. It worked. Far fewer planes were shot down and far more returned safely, ready to continue the war effort. Wald’s clear thinking helped win the war.” (pp. 51-52)
As a former resident of California’s Temecula Valley, I had studied some of the early urban planning studies of the area for my first book, The SimCity Planning Commission Handbook. So, I was very intrigued to read about how a mining company intended to ruin the “Rainbow Gap” (the secret to the valley’s cooling breeze which allowed for a respectable wine industry to grow there) and how they abused statistics to try to say their plan would have no environmental/economic negatives. For example, the mining company’s consultants argued that prices for homes next to their other mines still went up the same percentage as those miles away from one of their quarries. They didn’t compare values; they compared percentages (even though values had decreased by 20% since the quarry went in (p. 69).
As an inveterate coffee drinker, I was fascinated by both the historical reference to Gustav the Great’s experiment regarding coffee and two identical twins (convicted felons) (p. 154). Would you be surprised to realize that part of the problem with the conclusion was the small sample size and ignorance of confounding factors? Of course, Smith goes on to cite later studies which made similar errors because: “We consistently underestimate the role of chance in our lives, failing to recognize that randomness can generate patterns that appear to be meaningful, but are, in fact, meaningless.
We are too easily seduced by explanations for the inexplicable.” (p. 164)
If the result of ignoring the outliers on O-ring tests at NASA hadn’t been the deaths of the Challenger crew, it would have been amusing to read that Nobel prize-winner Richard Feynman demonstrated the significance of the cold-weather tests: “During nationally televised hearings, this Nobel Prize–winning theoretical physicist demonstrated that O-rings lost their resiliency at low temperatures simply by dunking one in a glass of ice water.” (pp. 237-238) Also, having dodged a bullet myself and observing the pain of those around me, I don’t find much amusement in the so-called “dot-bomb” era, but Smith does a solid job of explaining the misplaced logic behind it (pp. 311-314). I thought his summary in this regard was quite clever, too: “In carpentry, they say, ’Measure twice, cut once.’ With data, ’Think twice, calculate once.’” (p. 324).
In the final chapter, Smith recapitulates the errors he has delineated through the book. He reminds his readers: “Before you double-check someone’s arithmetic, double-check their reasoning.” (p. 360). And, though he is specifically addressing the “Texas sharpshooter” fallacy, there is general wisdom when he writes: “It is easy to find a theory that fits the data if the data are used to invent the theory.” (p. 363) Frankly, if more people read Standard Deviations: Flawed Assumptions, Tortured Data, and Other Ways to Lie with Statistics and applied its lessons, there would be a lot less ridiculous bestselling books on getting rich quick and many less gullible people falling for every dubious “scientific” study. This was not only a joy to read; it will be a book I return to on numerous occasions.
One gets a sense of where the volume is going from early on when author Gary Smith quotes Ronald Coase’s cynical comment: “If you torture the data long enough, it will confess.” (p. 19) I’d heard the quotation, but never knew the source. Then, after explaining the basis for our genetic predilection for pattern recognition (whether significant or not), the author states the problem with over relying on such an instinct: “Our inherited desire to explain what we see fuels two kinds of cognitive errors. First, we are too easily seduced by patterns and by the theories that explain them. Second, we latch onto data that support our theories and discount contradicting evidence.” (p. 21). Late in the book, Smith reiterates his warning: “Data without theory is alluring, but misleading.” (p. 307).
The beauty of the book is that it skewers the poor to abominable research practices in multiple fields: medicine, economics, sociology, parapsychology, military strategy, and even web design. His examples are derived from historical studies and modern studies, and his conclusions are clear. One’s mental alarm quickly signals what your visceral reaction to conflicting studies (particularly medical and psychological/sociological) over the years, to watch out. For example, he cites John Ionnadis of the Stanford University School of Medicine who: “looked at 45 of the most widely respected medical studies during the years 1990 through 2003 that claimed to have demonstrated effective treatments for various ailments. In only 34 cases were attempts made to replicate the original test results with larger samples. The initial results were confirmed in 20 of these 34 cases (59 percent). For seven treatments, the benefits were much smaller than initially estimated; for the other seven treatments, there were no benefits at all.” (p. 31)
In demonstrating problems, Smith cites: self-selection bias (p. 40), survivor bias (particularly in backward-looking studies—p. 56), confirmation bias (p. 68), deceptive graphing (“The ups and downs in the data can be exaggerated by putting a small range of numbers on the axis and can be dampened by putting a large range of numbers on the axis.”--p. 106), the false-positive problem (p. 136), confounding factors (p. 145), regression to the mean (p. 180), trust in the fallacious law of averages (p. 203), starting with data clusters rather than researching a theory (p. 213, otherwise known as the Texas sharpshooter approach – p. 215), be careful in discarding outliers (sometimes, you shouldn’t – p. 234), the selective reporting or “publication effect” (p. 273), use of forward or backward displacement to create a multiplicity of potentialities (p. 274), and extrapolation of past trends into future expectations (p. 288) seem to be the major pitfalls. In addition to these, Smith simply warned: “the two main pitfalls of quantitative financial analysis: a naive confidence that historical patterns are a reliable guide to the future, and a dependence on theoretical assumptions that are mathematically convenient but dangerously unrealistic.” (p. 331).
One of Smith’s most interesting examples to me (since I enjoy military history) was reading about a WWII effort to study damaged planes so that they would have a statistical basis for placing additional (albeit heavy) armor on vulnerable areas. Yet, the self-selecting fallacy in this study was that planes hit in the most vulnerable places like the cockpit and fuel lines didn’t usually make it home. Thanks to an analysis by Andrew Wald, authorities discovered: “Returning planes were more likely to have holes in the wings because these holes did little damage. Wald’s advice was exactly the opposite of the initial conclusion. Instead of reinforcing the locations with the most holes, they should reinforce the locations with no holes. It worked. Far fewer planes were shot down and far more returned safely, ready to continue the war effort. Wald’s clear thinking helped win the war.” (pp. 51-52)
As a former resident of California’s Temecula Valley, I had studied some of the early urban planning studies of the area for my first book, The SimCity Planning Commission Handbook. So, I was very intrigued to read about how a mining company intended to ruin the “Rainbow Gap” (the secret to the valley’s cooling breeze which allowed for a respectable wine industry to grow there) and how they abused statistics to try to say their plan would have no environmental/economic negatives. For example, the mining company’s consultants argued that prices for homes next to their other mines still went up the same percentage as those miles away from one of their quarries. They didn’t compare values; they compared percentages (even though values had decreased by 20% since the quarry went in (p. 69).
As an inveterate coffee drinker, I was fascinated by both the historical reference to Gustav the Great’s experiment regarding coffee and two identical twins (convicted felons) (p. 154). Would you be surprised to realize that part of the problem with the conclusion was the small sample size and ignorance of confounding factors? Of course, Smith goes on to cite later studies which made similar errors because: “We consistently underestimate the role of chance in our lives, failing to recognize that randomness can generate patterns that appear to be meaningful, but are, in fact, meaningless.
We are too easily seduced by explanations for the inexplicable.” (p. 164)
If the result of ignoring the outliers on O-ring tests at NASA hadn’t been the deaths of the Challenger crew, it would have been amusing to read that Nobel prize-winner Richard Feynman demonstrated the significance of the cold-weather tests: “During nationally televised hearings, this Nobel Prize–winning theoretical physicist demonstrated that O-rings lost their resiliency at low temperatures simply by dunking one in a glass of ice water.” (pp. 237-238) Also, having dodged a bullet myself and observing the pain of those around me, I don’t find much amusement in the so-called “dot-bomb” era, but Smith does a solid job of explaining the misplaced logic behind it (pp. 311-314). I thought his summary in this regard was quite clever, too: “In carpentry, they say, ’Measure twice, cut once.’ With data, ’Think twice, calculate once.’” (p. 324).
In the final chapter, Smith recapitulates the errors he has delineated through the book. He reminds his readers: “Before you double-check someone’s arithmetic, double-check their reasoning.” (p. 360). And, though he is specifically addressing the “Texas sharpshooter” fallacy, there is general wisdom when he writes: “It is easy to find a theory that fits the data if the data are used to invent the theory.” (p. 363) Frankly, if more people read Standard Deviations: Flawed Assumptions, Tortured Data, and Other Ways to Lie with Statistics and applied its lessons, there would be a lot less ridiculous bestselling books on getting rich quick and many less gullible people falling for every dubious “scientific” study. This was not only a joy to read; it will be a book I return to on numerous occasions.
November 30, 2017
Perhaps one of the books that gives critiques on the fallacy with regard to probabilities, statistics and the applied use of it in finance and investment, academia research, medical diagnosis and/or prognosis and a lot of other related fields that deals with data (sports science for instance).
Thus, the major part of this are dedicated to straighten the dividing line between correlation and causation.
As to the nature of this book as a critique, a lot of examples were drawn from an abundant of cases, thus perhaps at times makes this hard to read (imagine the graphs and numbers!)
A good (though somewhat troubling as of no reference) complement to Taleb's Black Swan and Fooled by Randomness
Thus, the major part of this are dedicated to straighten the dividing line between correlation and causation.
As to the nature of this book as a critique, a lot of examples were drawn from an abundant of cases, thus perhaps at times makes this hard to read (imagine the graphs and numbers!)
A good (though somewhat troubling as of no reference) complement to Taleb's Black Swan and Fooled by Randomness
January 30, 2022
The book takes the idea that we are predisposed to seeing patterns and identifying causal explanations for that pattern, despite how ludicrous those causes can be. Many of the errors involve misunderstanding how observed patterns in past data do not reveal causes unless retested predictively in fresh data. The earlier chapters present most of the underlying thesis and the later chapters provide further examples and nuanced representations of the earlier arguments. The final chapters were focused specifically on the stock market.
I'm just beginning my PhD journey, and I realised that none of my universities have yet demonstrated clearly how statistical validation is often manipulated or distorted, nor how it should be performed accurately. This book definitely presents a good general argument of how past observational data can be misleading or distorted. Although I found many of the points quite buried in the examples and arguments, which you often have to dig out, they are well-argued and supported. I particularly liked the comedic and cynical writing style of the book, with lots of analyses of popular books and studies. Favourite quote: "It doesn’t even pass the straight-face test."
Very glad to see some critical scrutiny toward popular studies and works, such as Jon Collins' Good to Great, and Peters and Waterman's In Search of Excellence. As a business researcher, I can't help but feel that such nonsense cheapens the whole field of management. I particularly love this quote:
When we look back in time at any group of companies, the best or the worst, we can always find some common characteristics. Look, every one of those eleven companies selected by Collins has either an i or an r in its name, and several have both an i and an r. Is the key for going from good to great to make sure that your company’s name has an i or r in it? (Chapter 2)
and also:
If it were so easy to predict the stock market, the gurus would get rich trading stocks instead of making a living by making predictions. (Chapter 13)
The only thing I think could be improved would be to include some common processes for researchers to avoid such errors and fallacies. But, with that said, it does provide a useful stepping stone for further research.
What did this book leave me with?
I'm just beginning my PhD journey, and I realised that none of my universities have yet demonstrated clearly how statistical validation is often manipulated or distorted, nor how it should be performed accurately. This book definitely presents a good general argument of how past observational data can be misleading or distorted. Although I found many of the points quite buried in the examples and arguments, which you often have to dig out, they are well-argued and supported. I particularly liked the comedic and cynical writing style of the book, with lots of analyses of popular books and studies. Favourite quote: "It doesn’t even pass the straight-face test."
Very glad to see some critical scrutiny toward popular studies and works, such as Jon Collins' Good to Great, and Peters and Waterman's In Search of Excellence. As a business researcher, I can't help but feel that such nonsense cheapens the whole field of management. I particularly love this quote:
When we look back in time at any group of companies, the best or the worst, we can always find some common characteristics. Look, every one of those eleven companies selected by Collins has either an i or an r in its name, and several have both an i and an r. Is the key for going from good to great to make sure that your company’s name has an i or r in it? (Chapter 2)
and also:
If it were so easy to predict the stock market, the gurus would get rich trading stocks instead of making a living by making predictions. (Chapter 13)
The only thing I think could be improved would be to include some common processes for researchers to avoid such errors and fallacies. But, with that said, it does provide a useful stepping stone for further research.
What did this book leave me with?
October 19, 2014
We are living in a golden age of pop math books, such as Jordan Ellenberg's How To Not Be Wrong and Ed Frenkel's Love And Math. Compared to those books, this one is quite disappointing. Which isn't to say it's entirely bad -- the author has compiled a large number of interesting anecdotes about people misusing statistics, and some of them have been the type of story I bring up at cocktail parties or mention to my students -- but overall the book lacks cohesion and doesn't hold together as well as I wanted.
June 25, 2016
Must read for all data analysts. The book gives insights into the data analytics , how it can be manipulate and influence big decisions.
December 1, 2021
Not a good audiobook! Might have been 3 stars if I had read it instead. Felt a bit ranty. Too many numbers and charts as an audiobook. Overall I kind of found the book annoying
April 29, 2023
The author does an excellent job of showing how statistics can be misinterpreted. He has great stories that illustrate his points and has produced explanations that are clear and compelling.
The earlier chapters are the strongest when he, with excellent examples, illustrates the concept of survivorship bias, regression toward the mean, and how people find patterns and tell stories to explain patterns that have been generated by random events.
There are so many great examples that author gives. Here are ten, in order of importance.
1. From Good to Best. The author shows why this book is based on statistical research that has zero value. It is such a good example of how a researcher said that he would let the data speak for themselves but instead demonstrated that he wasn't testing anything, that there was not any observation that would have caused him to be able to falsify any of his beliefs because he had no hypothesis.
2. World War II: protecting bombers. Where should the British put protective plates on bombers? One answer: look at where the bombers that came back had most holes from ground fire. The problem: that bombers that were hit in the more vulnerable areas (fuel lines) never came back.
3. Cats falling from high heights. Cats falling from higher heights had higher survival rates than cats falling from lower heights. The problem: fewer cats survived falls from high heights and therefore never showed up in the data.
4. Monty Hall example. This is quite involved but worth working through the example.
5. Probability of sex of other child.
6. Graduation rates at higher rated schools than lower rate schools.
7. Banning the sale of pitchers of beer would reduce consumption of beer.
8. Statistical evidence of discrimination.
9. Statistics showing that women at a graduate program are being discriminated against.
10. Power lines cause cancer.
The last three chapters of the book are not the strongest, almost a sense that the author wanted to achieve a minimum length of the paper but overall the author has wonderful examples, brings insight to topics others have covered, and uses real world examples to illustrate his points.
The earlier chapters are the strongest when he, with excellent examples, illustrates the concept of survivorship bias, regression toward the mean, and how people find patterns and tell stories to explain patterns that have been generated by random events.
There are so many great examples that author gives. Here are ten, in order of importance.
1. From Good to Best. The author shows why this book is based on statistical research that has zero value. It is such a good example of how a researcher said that he would let the data speak for themselves but instead demonstrated that he wasn't testing anything, that there was not any observation that would have caused him to be able to falsify any of his beliefs because he had no hypothesis.
2. World War II: protecting bombers. Where should the British put protective plates on bombers? One answer: look at where the bombers that came back had most holes from ground fire. The problem: that bombers that were hit in the more vulnerable areas (fuel lines) never came back.
3. Cats falling from high heights. Cats falling from higher heights had higher survival rates than cats falling from lower heights. The problem: fewer cats survived falls from high heights and therefore never showed up in the data.
4. Monty Hall example. This is quite involved but worth working through the example.
5. Probability of sex of other child.
6. Graduation rates at higher rated schools than lower rate schools.
7. Banning the sale of pitchers of beer would reduce consumption of beer.
8. Statistical evidence of discrimination.
9. Statistics showing that women at a graduate program are being discriminated against.
10. Power lines cause cancer.
The last three chapters of the book are not the strongest, almost a sense that the author wanted to achieve a minimum length of the paper but overall the author has wonderful examples, brings insight to topics others have covered, and uses real world examples to illustrate his points.
This entire review has been hidden because of spoilers.
July 15, 2023
“Our inherent desire to explain what we see fuels two kinds of cognitive errors. First, we’re too easily seduced by patterns and by the theories that explain them. Second, we latch onto data that support our theories and discount contradicting evidence.” Computers have allowed us to collect tons of data that can be sliced and spliced to prove must about any hair-brained theory you want. It is routine now to play with data first to find some kind of pattern and then devise a theory to fit it, rather than develop a theory and then use fresh data to test it. “…a theory – any theory---can inevitably be amassed by looking at lots of data and discarding data that don’t support the theory.”
I was aghast at some of the faulty research and flawed reasoning that has gone on that has impacted medicine, education, and government, costing people their jobs, health, and even lives. John Maynard Keynes noted that “Professional economists, after Malthus, were apparently unmoved by the lack of correspondence between the results of their theory and the facts of observation.” The same refusal to change thinking in the face of facts can be seen daily in the news.
The book totally supports my long-held contention that statistics should be required in high school, rather than algebra. Most people don’t use algebra as adults but everybody is exposed to statistics everyday and get duped by the so-called “experts,” because they don’t know how to interpret what they’re looking at. I would say this book should be required reading for congresspeople so they don’t get taken in by lobbyists and advisers armed with dubious data, but congresspeople would be more likely to look at this as a handbook for how to lie better to their constituents.
I liked that the author debunked Good to Great and In Search of Excellence (more like in search of cherry-picked data to create a theory to sell a book). He also panned Freakeconomics, whose author made the claim that legalized abortion reduces the crime rate.
Ignore confounding factors and regression to the mean to your peril! (Unless you want to publish a best-seller, that is).
I was aghast at some of the faulty research and flawed reasoning that has gone on that has impacted medicine, education, and government, costing people their jobs, health, and even lives. John Maynard Keynes noted that “Professional economists, after Malthus, were apparently unmoved by the lack of correspondence between the results of their theory and the facts of observation.” The same refusal to change thinking in the face of facts can be seen daily in the news.
The book totally supports my long-held contention that statistics should be required in high school, rather than algebra. Most people don’t use algebra as adults but everybody is exposed to statistics everyday and get duped by the so-called “experts,” because they don’t know how to interpret what they’re looking at. I would say this book should be required reading for congresspeople so they don’t get taken in by lobbyists and advisers armed with dubious data, but congresspeople would be more likely to look at this as a handbook for how to lie better to their constituents.
I liked that the author debunked Good to Great and In Search of Excellence (more like in search of cherry-picked data to create a theory to sell a book). He also panned Freakeconomics, whose author made the claim that legalized abortion reduces the crime rate.
Ignore confounding factors and regression to the mean to your peril! (Unless you want to publish a best-seller, that is).
May 12, 2023
I can appreciate statistical analyses which help the non-professional navigate stock-market investing an the data-based explanations for financial fluctuations (read the part about the 'flash crash' caused by the trading computers). There are also good chapters on how the the scientific method can be stood on one of its ears by researchers (data is mined for a theory that can describe it) or on its other (a theory is presented as fact without proper data and testing). People seem to be hard-wired to make sense of their world by looking for patterns, grouping phenomena, but all too often confusing causation and correlation. There are sections on ESP experiments and the lottery. Smith even takes on some of the fun ideas of Freakonomics, much to my chagrin (note that Freakonomics is much more readable, just saying). It would be interesting to read a follow-up volume by Smith, analyzing the creative ideas being spewed by politicians these days (Standard Deviations came out in 2014).
The main place Smith fails is in advising us to use "common sense" as a starting point in evaluating supposedly statistically-supported ideas. I'm sorry, but 'common sense' has been used to justify a lot of status quo social assortment, disenfranchisement and assumptions. A better way to approach the evaluation process is to invoke critical thinking processes, not an old saw spewed by those who benefit from acceptance of unjustified "norms".
The high point in my takeaways, though is a reanalysis of the data behind the Reinhart-Rogoff study, the current basis for angst about the US government debt. To summarize, these authors chose data that supported their conclusions and ignored data that didn't. So stop agonizing over the debt, except maybe to consider how much of the debt is due to tax cuts that haven't ever paid for themselves (my idea, not Smith's).
This book does not directly teach you how to "lie with statistics"; it shows you how other people have done so in the past or how people have been mislead by the numbers.
The main place Smith fails is in advising us to use "common sense" as a starting point in evaluating supposedly statistically-supported ideas. I'm sorry, but 'common sense' has been used to justify a lot of status quo social assortment, disenfranchisement and assumptions. A better way to approach the evaluation process is to invoke critical thinking processes, not an old saw spewed by those who benefit from acceptance of unjustified "norms".
The high point in my takeaways, though is a reanalysis of the data behind the Reinhart-Rogoff study, the current basis for angst about the US government debt. To summarize, these authors chose data that supported their conclusions and ignored data that didn't. So stop agonizing over the debt, except maybe to consider how much of the debt is due to tax cuts that haven't ever paid for themselves (my idea, not Smith's).
This book does not directly teach you how to "lie with statistics"; it shows you how other people have done so in the past or how people have been mislead by the numbers.
July 5, 2019
The first half of the book held my interest as each chapter contained real-life examples of how statistics are misused, how people have based bad decisions on information that was presented incorrectly and how bad information can be made to look perfectly reasonable. There is nothing here for anyone who has any statistics background, but it seems like an entertaining introduction for the rest of us.
There are some errors described in the book that I see almost every day. For example, Chapter 2 is about self-selection bias and survivor bias. After reading the chapter and looking at the graphics, it is difficult to understand how so many people get it wrong. "Theory without data" was one of my pet peeves at work. I was often told "I want to use _____ as an example in my term paper/lecture/book. Find me something that proves it."
However, I thought that the most interesting anecdotes were used in the first half of the book and that the end of the book dragged.
Nit picking -- On page 60 the author says that researchers made an "error" when data that contradicted their conclusions was available but omitted from their calculations. As an "error" the example is suitable for inclusion in this chapter but I don't see what data he has to support the conclusion that it was an error rather than cheating.
There are some errors described in the book that I see almost every day. For example, Chapter 2 is about self-selection bias and survivor bias. After reading the chapter and looking at the graphics, it is difficult to understand how so many people get it wrong. "Theory without data" was one of my pet peeves at work. I was often told "I want to use _____ as an example in my term paper/lecture/book. Find me something that proves it."
However, I thought that the most interesting anecdotes were used in the first half of the book and that the end of the book dragged.
Nit picking -- On page 60 the author says that researchers made an "error" when data that contradicted their conclusions was available but omitted from their calculations. As an "error" the example is suitable for inclusion in this chapter but I don't see what data he has to support the conclusion that it was an error rather than cheating.
Displaying 1 - 30 of 107 reviews