What do you think?
Rate this book
Bad Data: Why We Measure the Wrong Things and Often Miss the Metrics That Matter
Highlights the pitfalls of data analysis and emphasizes the importance of using the appropriate metrics before making key decisions.
Big data is often touted as the key to understanding almost every aspect of contemporary life. This critique of "information hubris" shows that even more important than data is finding the right metrics to evaluate it.
The author, an expert in environmental design and city planning, examines the many ways in which we measure ourselves and our world. He dissects the metrics we apply to health, worker productivity, our children's education, the quality of our environment, the effectiveness of leaders, the dynamics of the economy, and the overall well-being of the planet. Among the areas where the wrong metrics have led to poor outcomes, he cites the fee-for-service model of health care, corporate cultures that emphasize time spent on the job while overlooking key productivity measures, overreliance on standardized testing in education to the detriment of authentic learning, and a blinkered focus on carbon emissions, which underestimates the impact of industrial damage to our natural world.
He also examines various communities and systems that have achieved better outcomes by adjusting the ways in which they measure data. The best results are attained by those that have learned not only what to measure and how to measure it, but what it all means.
By highlighting the pitfalls inherent in data analysis, this illuminating book reminds us that not everything that can be counted really counts.
Big data is often touted as the key to understanding almost every aspect of contemporary life. This critique of "information hubris" shows that even more important than data is finding the right metrics to evaluate it.
The author, an expert in environmental design and city planning, examines the many ways in which we measure ourselves and our world. He dissects the metrics we apply to health, worker productivity, our children's education, the quality of our environment, the effectiveness of leaders, the dynamics of the economy, and the overall well-being of the planet. Among the areas where the wrong metrics have led to poor outcomes, he cites the fee-for-service model of health care, corporate cultures that emphasize time spent on the job while overlooking key productivity measures, overreliance on standardized testing in education to the detriment of authentic learning, and a blinkered focus on carbon emissions, which underestimates the impact of industrial damage to our natural world.
He also examines various communities and systems that have achieved better outcomes by adjusting the ways in which they measure data. The best results are attained by those that have learned not only what to measure and how to measure it, but what it all means.
By highlighting the pitfalls inherent in data analysis, this illuminating book reminds us that not everything that can be counted really counts.
323 pages, Hardcover
First published January 1, 2019
28 people are currently reading
212 people want to read
Ratings & Reviews
Friends & Following
Create a free account to discover what your friends think of this book!
Community Reviews
Displaying 1 - 24 of 24 reviews
January 29, 2020
First, let me clearly state that I am not a forgiving reader. Now that we have that out of the way...
I was really looking forward to this book. I waited for several months while my library bought and processed the book. So imagine my utter dismay when I started reading chapter one and, on the first page, came across an inaccurate factual statement. An easily verified factual statement that this author simply didn't bother to verify. It stopped me dead in my tracks.
(Edited to clarify: The author stated that the Common Core was "...developed by the Obama administration...." The Common Core was not developed by the Obama administration, it was a state effort begun by a consensus of state governors. This fact is easily verified by a simple online search. I happen to know that this statement by the author was incorrect but it made me wonder what other simple facts the author got wrong; therefore, I could no longer trust anything I read.)
So after a week of convincing myself to keep reading, I continued with chapter one. This chapter is titled Teaching to the Test. I am a huge advocate of NOT teaching to the test. I have worked in schools, my friend works in a public school, someone close to me worked as a tutor for high school students preparing for their exams in another country. I have strong feelings about this subject. And I was fully prepared to agree with the author that teaching to the test, and then relying exclusively on that testing score, is the wrong way to go about measuring and using data.
He starts the chapter talking about the Common Core (which is where he makes his inaccurate statement) and he tells of a well-regarded person who kills herself after it comes out that she cheated to get her students to do well on her schools exams. (Since this is a non-fiction book, I don't consider any of this information spoilers. My apologies if you do.) So his viewpoint about the negative effects of this testing system are clear.
But then the author gets sloppy again. He states "Every year, high school seniors all over the world prepare for their final exams." Now yes, he said "all over the world" but he also said "high school seniors" which is part of the American system and he just finished discussing the Common Core, which is exclusive to America. So I think it likely that the reader will continue to read as if the author were discussing America. He continues, "Take the math final exam, for example: a three hour marathon of mathematical problem-solving." The only students who take a 3 hour one-subject test in America is an AP student, and that test is for college credit so yes, it is longer and more intensive. If a student is not an AP high school senior in America, that student is not taking a three-hour, one subject exam. Students in other countries, however, do take three hour long (and perhaps longer) exams in one subject area as a regular part of their A level exams. But the author is not clear about which country or countries he is discussing. Then he mentions the Common Core again, which once again puts the reader squarely back in America.
Finally, he states, "Sadly, it is becoming familiar too to those in much younger grade levels; standardized tests are now part of the curriculum for students as early as third grade in Common Core states."
I took standardized tests in school when I was in third grade. That was in 1973. This author is equating standardized tests with the Common Core, another factual inaccuracy. Standardized tests are given in every school in every state regardless of whether they are a Common Core state or not. And standardized tests have been given in schools as young as the third grade for decades. The difference is, back then, we took a standardized test in 3rd, 5th and 7th grade. And the score on the test was not used exclusively to determine too much of anything. Now, students have to take standardized tests at every grade level, sometimes multiple times a year, and the score is highly determinative for many things.
My point is this: in the 4, yes only 4, pages I have read, this author gives inaccurate factual statements and conflates information into one generalized viewpoint. If the author manages this much poor writing in 4 pages of his book about data, then I am not going to bother reading the rest of the book. The irony is that, in the prologue, the author states that the reason we use and rely on data, or metrics, his preferred word, is: "Metrics provide an objective, dispassionate, and consistent criteria that we can use to compare and evaluate performance. Metrics allows us to remove the messy, sometimes controversial and emotionally charged, discussions about what is important and why."
Well, maybe metrics can do all that but this author obviously cannot because he is not careful in stating his facts and making clear distinctions in his writing. I expect much more in a non-fiction book.
I was really looking forward to this book. I waited for several months while my library bought and processed the book. So imagine my utter dismay when I started reading chapter one and, on the first page, came across an inaccurate factual statement. An easily verified factual statement that this author simply didn't bother to verify. It stopped me dead in my tracks.
(Edited to clarify: The author stated that the Common Core was "...developed by the Obama administration...." The Common Core was not developed by the Obama administration, it was a state effort begun by a consensus of state governors. This fact is easily verified by a simple online search. I happen to know that this statement by the author was incorrect but it made me wonder what other simple facts the author got wrong; therefore, I could no longer trust anything I read.)
So after a week of convincing myself to keep reading, I continued with chapter one. This chapter is titled Teaching to the Test. I am a huge advocate of NOT teaching to the test. I have worked in schools, my friend works in a public school, someone close to me worked as a tutor for high school students preparing for their exams in another country. I have strong feelings about this subject. And I was fully prepared to agree with the author that teaching to the test, and then relying exclusively on that testing score, is the wrong way to go about measuring and using data.
He starts the chapter talking about the Common Core (which is where he makes his inaccurate statement) and he tells of a well-regarded person who kills herself after it comes out that she cheated to get her students to do well on her schools exams. (Since this is a non-fiction book, I don't consider any of this information spoilers. My apologies if you do.) So his viewpoint about the negative effects of this testing system are clear.
But then the author gets sloppy again. He states "Every year, high school seniors all over the world prepare for their final exams." Now yes, he said "all over the world" but he also said "high school seniors" which is part of the American system and he just finished discussing the Common Core, which is exclusive to America. So I think it likely that the reader will continue to read as if the author were discussing America. He continues, "Take the math final exam, for example: a three hour marathon of mathematical problem-solving." The only students who take a 3 hour one-subject test in America is an AP student, and that test is for college credit so yes, it is longer and more intensive. If a student is not an AP high school senior in America, that student is not taking a three-hour, one subject exam. Students in other countries, however, do take three hour long (and perhaps longer) exams in one subject area as a regular part of their A level exams. But the author is not clear about which country or countries he is discussing. Then he mentions the Common Core again, which once again puts the reader squarely back in America.
Finally, he states, "Sadly, it is becoming familiar too to those in much younger grade levels; standardized tests are now part of the curriculum for students as early as third grade in Common Core states."
I took standardized tests in school when I was in third grade. That was in 1973. This author is equating standardized tests with the Common Core, another factual inaccuracy. Standardized tests are given in every school in every state regardless of whether they are a Common Core state or not. And standardized tests have been given in schools as young as the third grade for decades. The difference is, back then, we took a standardized test in 3rd, 5th and 7th grade. And the score on the test was not used exclusively to determine too much of anything. Now, students have to take standardized tests at every grade level, sometimes multiple times a year, and the score is highly determinative for many things.
My point is this: in the 4, yes only 4, pages I have read, this author gives inaccurate factual statements and conflates information into one generalized viewpoint. If the author manages this much poor writing in 4 pages of his book about data, then I am not going to bother reading the rest of the book. The irony is that, in the prologue, the author states that the reason we use and rely on data, or metrics, his preferred word, is: "Metrics provide an objective, dispassionate, and consistent criteria that we can use to compare and evaluate performance. Metrics allows us to remove the messy, sometimes controversial and emotionally charged, discussions about what is important and why."
Well, maybe metrics can do all that but this author obviously cannot because he is not careful in stating his facts and making clear distinctions in his writing. I expect much more in a non-fiction book.
dnf
September 6, 2024DNF @ page 72
Started off interestingly, but I don't think I really liked the direction it went in. Also, after the first two chapters, it got incredibly dry and boring.
Started off interestingly, but I don't think I really liked the direction it went in. Also, after the first two chapters, it got incredibly dry and boring.
October 14, 2024
I opened this book hoping to learn how to better use data and personal goals to improve my life, and to apply a better understanding of data towards improving the world. It's a fairly accurate book without obvious mistakes, but I did not benefit from reading it.
The author is a fine storyteller, of many stories that I have heard before. If you are seeking a book to learn about data analysis in even a quasi academic format, this isn't it.
At the end of chapter 9, the author summarizes the learnings from this book in 14 enumerated paragraphs. It is an accurate summary, and just reinforced my disappointment. Because the author is such a good storyteller, I engaged with many of the stories, even though I had heard them before, and this was just a retelling of my memory, the summary reminded me that I had gained next to nothing from this book.
But I have a background and social science and data analysis, so perhaps I'm being too harsh, yet then again, I imagine that I am within his core target audience for this book.
I will highlight that in chapter 8 he cites Daniel Ariely, unironically, even though he has been accused of fabricating data in his dishonesty experiments with Harvard's Gino. The entire book under emphasizes dishonesty in experimental data, even as it repeatedly fines, dishonesty, motivated by incentives during real life.
The author is a fine storyteller, of many stories that I have heard before. If you are seeking a book to learn about data analysis in even a quasi academic format, this isn't it.
At the end of chapter 9, the author summarizes the learnings from this book in 14 enumerated paragraphs. It is an accurate summary, and just reinforced my disappointment. Because the author is such a good storyteller, I engaged with many of the stories, even though I had heard them before, and this was just a retelling of my memory, the summary reminded me that I had gained next to nothing from this book.
But I have a background and social science and data analysis, so perhaps I'm being too harsh, yet then again, I imagine that I am within his core target audience for this book.
I will highlight that in chapter 8 he cites Daniel Ariely, unironically, even though he has been accused of fabricating data in his dishonesty experiments with Harvard's Gino. The entire book under emphasizes dishonesty in experimental data, even as it repeatedly fines, dishonesty, motivated by incentives during real life.
January 16, 2021
Schryvers does an excellent job of what could (on paper) be a very dry subject. What gets measured, what decisions are made as a result of all those measures is critical knowledge that helps us measure what really counts and what’s really important in life. Hint: it’s not necessarily the GDP.
June 12, 2023
Great content, poorly edited. Extremely repetitive writing. The book could've been half it's size
January 6, 2022
A tad wordy and round-about (things could have been phrased more succinctly), but the underlying message is clear - metrics may be objective, but choosing what to measure and how to measure is most definitely a subjective value judgment.
___
The 'ant mill' - where ants follow their pheromone trails in a never ending circle until they all die from exhaustion. Biological feedback l00p.
Metrics help us understand the truth - intuition can be wildly wrong at times.
Metrics help reduce complex system in legible and meaningful ways, providing us with simplicity (when chosen well).
Metrics address issues of trust. Most people would say that they are better than their peers. These separate criteria can be used to verify claims.
They are also objective, though the selection of metrics is in itself subjective. The goal of many metrics is to improve what we do, by telling us where we are, and in which direction we should strive.
MCQ tests lead students to the belief that being smart is simply a matter of knowing a lot of facts and being able to remember things quickly (quiz show intelligence). They are often little more than a measurement of someone's short-term memory.
MCQs also lack an important component of a good test - requiring the student to generate the answer themselves.
Goodhart's Law - when a measure becomes a target, it ceases to be a good measure. When a measure is tied to incentives, people will find a way to maximise that measure, whether or not their actions help achieve the original intent of the metric (see breeding cobras / centipedes for bounties, or 19th century Chinese breaking up dinosaur bones which were rewarded per piece).
Often metrics are chosen because they are easy to measure, not because they are good indicators.
What matters most in the healthcare system is not techniques used or equipment available (though important), but something less tangible - trust. When patients trust their doctor, they listen to doctor's advice and follow instructions. Lack of trust leads to unnecessary testing.
Canada's CVFP had a pay-per-patient model, which incentivised them to keep their patient roster healthy so they could handle more. This is vulnerable to 'cream skimming', where the clinic rejects unhealthy patients because they would be burdensome.
To ensure standard of care, 'negations', or the clinic being billed if their patient sought treatment elsewhere, were put in place.
Input metrics can be misused by emphasising how large/small they are, like money spent per patient/amount of budget cut even if the link between input and outcomes is not well understood.
In the world of fund managers, the power of the herd is strong, and that herd is short-sighted. There are many ways to respond to pressure for short-term earnings - fudge the numbers, or sacrifice future capability (e.g. slash marketing / R&D) for today's balance sheet.
The lemon problem - if there is no way to accurately discern between high and low quality products, consumers will simply assume all products are of low quality. High quality products cannot charge a premium and suffer a loss.
Firms with large institutional investors show a lower probability of cutting R&D spending.
Bibliometrics in the field of science (chasing papers and citations) hurts the general progress of science, as scientists focus on writing grant proposals, and universities shift into profit centres focused on creating new products or patents, rather than providing science as a public good, or making discoveries that may not have immediate commercial application. We cannot predict ahead of time the impact or success of a research study.
It is counter-intuitively better to provide low-powered incentives, such as basic salaries rather than bonuses to scientists for research outcomes. With fundamental research, it is better to ensure that inputs are optimised, because it is long-term and high-risk.
Funding agencies should know that they cannot predict the outcomes or timing of fundamental research. What they can do is ensure the environment is enabling.
If the mortality of a disease drops, its prevalence will rise, as more people survive each year to be counted as living with the disease (e.g. malaria). It is better to look at the burden of disease, but that is harder to measure.
Banks, when advising mortgages, do not take into account location (affecting commuting time), even though transportation is the 2nd-largest household expenditure (in Canada, at 20.4%). This is higher than healthcare, education and food combined (granted most education and healthcare is paid by the state through taxes).
Optimising each component doesn't necessarily lead to optimisation of the whole. E.g. offshoring for low-cost manufacturing also increasing manufacturing lead time, which could lead to longer time to market, or more unsold inventory at the end of the season (Zara, fast fashion).
Food miles doesn't make sense because food is shipped around the world in bulk freight. The largest carbon impact comes from the drive to purchase the food at the supermarket, which would account for 48% of all transportation miles associated with food. Even the Food Miles Report notes that it is usually more energy efficient to grow food at its natural climatic conditions and ship them, rather than energy intensive greenhouses.
Metrics such as plus/minus (assessing the score difference before and after a basketball player is playing on the court) help to eliminate noise from multiple factors (teamwork, individual skill) influencing a single outcome (the scoreline). It assumes that there are many hidden, misunderstood or undervalued contributions a player makes that can help win a game.
Disability adjusted life years (DALYs) help account for the burden of disease. A severe depressive episode is rated above blindness, Alzheimer's and even a spinal cord injury.
Organisations that focus on counting quantity while ignoring quality will soon see their workers doing 'cream skimming' - only counting the good stuff while ignoring, avoiding or neglecting the rest.
To prevent cream skimming, one could set up minimum standards, but perversely this will discourage effort beyond the minimum. It might also encourage people to give up if the project doesn't have a good chance of meeting the standard.
Whistle-blowers tend to be those with nothing to lose, and thus the management tools used to pressure most people (threat of job loss) don't work as well on them.
Right from the beginning of American involvement in Vietnam, there were issues establishing a system of measurement for the war. Initially there was a flood of data, with 14000 pounds of reports produced daily.
Eventually the strategy became attrition, the chief metric was decided to be body count, to the detriment of troops' lives, which were expended unnecessarily counting bodies.
Like so many managers who overemphasise numbers and neglect everything else in their jobs, McNamara used data to mask the fact that he couldn't comprehend the nature of the conflict, the psychology and motivations of the Vietnamese, or the complex political, cultural and sociological factors influencing the war.
Social norms versus market norms - asking for help as a friend is a social norm. Paying you for help is a market norm, and the calculus involved is very different. Plus, they don't mix. Paying people to do chores crowds out the friendship and loyalty motivation.
Aversion to subjectivity drives people towards 'objective' metrics. But the choosing of what metric to use is itself subjective, a value judgement. We have to think hard about the metrics we choose and defend why we use them.
Trust, or the lack thereof, is at the root of nearly every measure we use when dealing with people. Virtually every performance standard, productivity report, activity requirement, and work evaluation is grounded in a basic, but unstated belief: We do not trust one another. The need for certainty stems out of a lack of trust. (The irony of "trust, but verify" is apparent).
Khan Academy's founder's approach to learning: You are not expected to know every concept, but you are expected to want to learn it. Tests become the means to the end that is learning. If you are not interested in ranking students, the idea of only taking a test once becomes silly.
___
The 'ant mill' - where ants follow their pheromone trails in a never ending circle until they all die from exhaustion. Biological feedback l00p.
Metrics help us understand the truth - intuition can be wildly wrong at times.
Metrics help reduce complex system in legible and meaningful ways, providing us with simplicity (when chosen well).
Metrics address issues of trust. Most people would say that they are better than their peers. These separate criteria can be used to verify claims.
They are also objective, though the selection of metrics is in itself subjective. The goal of many metrics is to improve what we do, by telling us where we are, and in which direction we should strive.
MCQ tests lead students to the belief that being smart is simply a matter of knowing a lot of facts and being able to remember things quickly (quiz show intelligence). They are often little more than a measurement of someone's short-term memory.
MCQs also lack an important component of a good test - requiring the student to generate the answer themselves.
Goodhart's Law - when a measure becomes a target, it ceases to be a good measure. When a measure is tied to incentives, people will find a way to maximise that measure, whether or not their actions help achieve the original intent of the metric (see breeding cobras / centipedes for bounties, or 19th century Chinese breaking up dinosaur bones which were rewarded per piece).
Often metrics are chosen because they are easy to measure, not because they are good indicators.
What matters most in the healthcare system is not techniques used or equipment available (though important), but something less tangible - trust. When patients trust their doctor, they listen to doctor's advice and follow instructions. Lack of trust leads to unnecessary testing.
Canada's CVFP had a pay-per-patient model, which incentivised them to keep their patient roster healthy so they could handle more. This is vulnerable to 'cream skimming', where the clinic rejects unhealthy patients because they would be burdensome.
To ensure standard of care, 'negations', or the clinic being billed if their patient sought treatment elsewhere, were put in place.
Input metrics can be misused by emphasising how large/small they are, like money spent per patient/amount of budget cut even if the link between input and outcomes is not well understood.
In the world of fund managers, the power of the herd is strong, and that herd is short-sighted. There are many ways to respond to pressure for short-term earnings - fudge the numbers, or sacrifice future capability (e.g. slash marketing / R&D) for today's balance sheet.
The lemon problem - if there is no way to accurately discern between high and low quality products, consumers will simply assume all products are of low quality. High quality products cannot charge a premium and suffer a loss.
Firms with large institutional investors show a lower probability of cutting R&D spending.
Bibliometrics in the field of science (chasing papers and citations) hurts the general progress of science, as scientists focus on writing grant proposals, and universities shift into profit centres focused on creating new products or patents, rather than providing science as a public good, or making discoveries that may not have immediate commercial application. We cannot predict ahead of time the impact or success of a research study.
It is counter-intuitively better to provide low-powered incentives, such as basic salaries rather than bonuses to scientists for research outcomes. With fundamental research, it is better to ensure that inputs are optimised, because it is long-term and high-risk.
Funding agencies should know that they cannot predict the outcomes or timing of fundamental research. What they can do is ensure the environment is enabling.
If the mortality of a disease drops, its prevalence will rise, as more people survive each year to be counted as living with the disease (e.g. malaria). It is better to look at the burden of disease, but that is harder to measure.
Banks, when advising mortgages, do not take into account location (affecting commuting time), even though transportation is the 2nd-largest household expenditure (in Canada, at 20.4%). This is higher than healthcare, education and food combined (granted most education and healthcare is paid by the state through taxes).
Optimising each component doesn't necessarily lead to optimisation of the whole. E.g. offshoring for low-cost manufacturing also increasing manufacturing lead time, which could lead to longer time to market, or more unsold inventory at the end of the season (Zara, fast fashion).
Food miles doesn't make sense because food is shipped around the world in bulk freight. The largest carbon impact comes from the drive to purchase the food at the supermarket, which would account for 48% of all transportation miles associated with food. Even the Food Miles Report notes that it is usually more energy efficient to grow food at its natural climatic conditions and ship them, rather than energy intensive greenhouses.
Metrics such as plus/minus (assessing the score difference before and after a basketball player is playing on the court) help to eliminate noise from multiple factors (teamwork, individual skill) influencing a single outcome (the scoreline). It assumes that there are many hidden, misunderstood or undervalued contributions a player makes that can help win a game.
Disability adjusted life years (DALYs) help account for the burden of disease. A severe depressive episode is rated above blindness, Alzheimer's and even a spinal cord injury.
Organisations that focus on counting quantity while ignoring quality will soon see their workers doing 'cream skimming' - only counting the good stuff while ignoring, avoiding or neglecting the rest.
To prevent cream skimming, one could set up minimum standards, but perversely this will discourage effort beyond the minimum. It might also encourage people to give up if the project doesn't have a good chance of meeting the standard.
Whistle-blowers tend to be those with nothing to lose, and thus the management tools used to pressure most people (threat of job loss) don't work as well on them.
Right from the beginning of American involvement in Vietnam, there were issues establishing a system of measurement for the war. Initially there was a flood of data, with 14000 pounds of reports produced daily.
Eventually the strategy became attrition, the chief metric was decided to be body count, to the detriment of troops' lives, which were expended unnecessarily counting bodies.
Like so many managers who overemphasise numbers and neglect everything else in their jobs, McNamara used data to mask the fact that he couldn't comprehend the nature of the conflict, the psychology and motivations of the Vietnamese, or the complex political, cultural and sociological factors influencing the war.
Social norms versus market norms - asking for help as a friend is a social norm. Paying you for help is a market norm, and the calculus involved is very different. Plus, they don't mix. Paying people to do chores crowds out the friendship and loyalty motivation.
Aversion to subjectivity drives people towards 'objective' metrics. But the choosing of what metric to use is itself subjective, a value judgement. We have to think hard about the metrics we choose and defend why we use them.
Trust, or the lack thereof, is at the root of nearly every measure we use when dealing with people. Virtually every performance standard, productivity report, activity requirement, and work evaluation is grounded in a basic, but unstated belief: We do not trust one another. The need for certainty stems out of a lack of trust. (The irony of "trust, but verify" is apparent).
Khan Academy's founder's approach to learning: You are not expected to know every concept, but you are expected to want to learn it. Tests become the means to the end that is learning. If you are not interested in ranking students, the idea of only taking a test once becomes silly.
December 16, 2022
Highly recommend. Worth buying.
November 14, 2021
Informative, enjoyable and has twisted my thinking (as books should). I was ever so slightly annoyed by some grammatical errors and (what I perceive as) subpar editing, but it was still a great read that I have recommended to others.
July 13, 2024
I maintain that this book would put a stake in the heart of any MBA if MBAs had souls to begin with. You can tell by some of the reviews that people become impatient with Schryvers which is super ironic
January 31, 2021
Good content worth exploring but the author did not structure his book well. Lots of repetition, verbal diarrhea, and I think his editor skipped town. Pity bc the underlying message is important.
August 26, 2023
Bad Data - Bad Book
I picked up this book as I thought it may have some interesting insights on some of the dangers presented by the use of data in today’s data analytics driven world. For example, the risks of mistaking correlation for causation can lead to poor decision making. This book adopts a more simplistic approach by discussing mundane areas where traditional methods of measuring a specific objective do not necessarily result in the optimum outcome - such as using multi-choice tests to determine college entrance or measuring worker productivity based on hours worked rather than output. In such cases, it is not so much that the data is bad as the inherent constraints (such as cost and time) of the process are sub-optimal.
Social media giants today use a range of data tracking and analysis techniques, such as tracking cookies, analyzing likes and tag suggestions that are designed to analyze human behavior but, in my estimation, are still somewhat naive and intrusive and lead to invalid assumptions about my interests and intent. Because I once checked out women’s apparel for my wife’s birthday or a rap record for the child of a friend does not make me permanently interested in such products. More intelligent analysis of my persona and interests would result in more targeted product offers. In short, analyzing where Facebook, Google and other sites get it wrong would be more informative than some interesting, but ultimately unsatisfying, stories about users misusing metrics.
I picked up this book as I thought it may have some interesting insights on some of the dangers presented by the use of data in today’s data analytics driven world. For example, the risks of mistaking correlation for causation can lead to poor decision making. This book adopts a more simplistic approach by discussing mundane areas where traditional methods of measuring a specific objective do not necessarily result in the optimum outcome - such as using multi-choice tests to determine college entrance or measuring worker productivity based on hours worked rather than output. In such cases, it is not so much that the data is bad as the inherent constraints (such as cost and time) of the process are sub-optimal.
Social media giants today use a range of data tracking and analysis techniques, such as tracking cookies, analyzing likes and tag suggestions that are designed to analyze human behavior but, in my estimation, are still somewhat naive and intrusive and lead to invalid assumptions about my interests and intent. Because I once checked out women’s apparel for my wife’s birthday or a rap record for the child of a friend does not make me permanently interested in such products. More intelligent analysis of my persona and interests would result in more targeted product offers. In short, analyzing where Facebook, Google and other sites get it wrong would be more informative than some interesting, but ultimately unsatisfying, stories about users misusing metrics.
October 12, 2025
3 stars, mostly because I've already learned about most of what the author has covered in my readings on metrics, Taylorism, Theory of Constraints, 6-Sigma, etc.
My main problem is that while the book is excellent at explaining how metrics can be designed and used poorly; indeed in some cases, downright wrongly. I don't share his somewhat nihilistic view that metrics are a dead end. There is one short-ish chapter on how to handle metrics with care, but nothing at all about how you would go about designing a solid metric (GQIM), and how to ensure that the use of that metric doesn't get perverted over time. Let's face it, metrics aren't going anywhere, so having some material, or at least pointing to other material, that would help us make better use would be tremendously useful.
I will chalk up most of what the author is complaining about to Goodhart's Law. What isn't directly covered by this, can be further summed up by the line from George Box "All models are wrong; some are useful." Meaning when you collapse a complicated system into a few metrics to measure, you've built a model. Is this model useful, and to what extent? Metrics are a bit like a knife that can twist in your hand and cut the wielder if you're not careful.
A good book if you're encountering for the first time that metrics can often go wrong. If you want to go beyond the adage of "lies, damned lies, and statistics" and really try to understand, start with this book, but don't stop with this book.
My main problem is that while the book is excellent at explaining how metrics can be designed and used poorly; indeed in some cases, downright wrongly. I don't share his somewhat nihilistic view that metrics are a dead end. There is one short-ish chapter on how to handle metrics with care, but nothing at all about how you would go about designing a solid metric (GQIM), and how to ensure that the use of that metric doesn't get perverted over time. Let's face it, metrics aren't going anywhere, so having some material, or at least pointing to other material, that would help us make better use would be tremendously useful.
I will chalk up most of what the author is complaining about to Goodhart's Law. What isn't directly covered by this, can be further summed up by the line from George Box "All models are wrong; some are useful." Meaning when you collapse a complicated system into a few metrics to measure, you've built a model. Is this model useful, and to what extent? Metrics are a bit like a knife that can twist in your hand and cut the wielder if you're not careful.
A good book if you're encountering for the first time that metrics can often go wrong. If you want to go beyond the adage of "lies, damned lies, and statistics" and really try to understand, start with this book, but don't stop with this book.
October 15, 2025
A deeply profound book that is a must read for anyone in data science, data analytics, or business intelligence. It goes beyond the goal of conventional statistics of analyzing data that is accurate or true and challenges us to examine whether the data being analyzed is relevant or matters at all.
So many companies and organizations focus on metrics and KPIs but how many of them choose metrics primarily because they can be easily and accurately measured? “An accurate measurement isn’t the same thing as a good one.”
The author is not critical of metrics, but urges caution on how they are used. While metrics are essential, it is important for leaders to understand the difference between measuring inputs, activities, outputs, and outcomes and how focusing on what’s easy to measure can can cause organizations to forget about the outcomes altogether.
We can’t forget that the important thing is the outcome, not the metric. The goal is the outcome, not the metric. Forgetting this results in perversion of what the metrics are supposed to accomplish or outright manipulation or fabrication of data. Not everything that is counted counts and not everything that counts can be counted. Measurements and data have their usefulness but it’s important to understand the limitations of attempting to quantify the things that are qualitative.
So many companies and organizations focus on metrics and KPIs but how many of them choose metrics primarily because they can be easily and accurately measured? “An accurate measurement isn’t the same thing as a good one.”
The author is not critical of metrics, but urges caution on how they are used. While metrics are essential, it is important for leaders to understand the difference between measuring inputs, activities, outputs, and outcomes and how focusing on what’s easy to measure can can cause organizations to forget about the outcomes altogether.
We can’t forget that the important thing is the outcome, not the metric. The goal is the outcome, not the metric. Forgetting this results in perversion of what the metrics are supposed to accomplish or outright manipulation or fabrication of data. Not everything that is counted counts and not everything that counts can be counted. Measurements and data have their usefulness but it’s important to understand the limitations of attempting to quantify the things that are qualitative.
July 6, 2023
As an upcoming college freshman planning studying data science, this was a transformative book for me. While my overstimulated teenage brain struggled to focus through some of the long examples and stories, I retained all of the main arguments from each chapter. I’m more cognizant of both the things I regularly measure and those metrics we all commonly hear and use.
I’ve never been more excited to continue my education and dive deeper into the world of data at university. Speaking of, many of my suspicions about and criticisms of high school education have been elaborated and supported in many ways. I was particularly interested in all the material on education and testing and I have a new perspective heading into college.
I’ve never been more excited to continue my education and dive deeper into the world of data at university. Speaking of, many of my suspicions about and criticisms of high school education have been elaborated and supported in many ways. I was particularly interested in all the material on education and testing and I have a new perspective heading into college.
March 30, 2025
Uneven. Some chapters are excellent while others..... not so much. Repetitive. The author has a habit of making a point then repeating it in the same paragraph then repeating it later in the chapter then later again in the following chapters. Chapter 1 is especially bad and almost convinced me to ditch this book. Dead horses. Schryvers never saw a dead horse that he did not need to flog. Schryvers has a few basic ideas that he introduces in the first chapter and then he proceeds to beat them to death for the next 300 pages. A competent editor could easily cut 100 pages from this book.
Read
January 24, 2021(I know it's just a catchy title but I feel bad for the data, the data didn't do anything! People used data poorly!) Anyway, read further for a number of interesting examples for how metrics can used and incentivized poorly and produce outcomes that don't fit with what we were hoping for. It's a data analytics version of "don't mistake the map for the territory" when complex or challenging systems are simplified to their detriment.
November 2, 2020
I'm two chapters in and already loving it - I can see some stuff about measuring outcomes rather than outputs (a distinction I now *actually* understand rather than a guess) and applying it to my job.
Written in a breezy, conversational manner. Really easy to kick back with a chapter or two at night. Love it!
Written in a breezy, conversational manner. Really easy to kick back with a chapter or two at night. Love it!
March 4, 2025
Great start, but goes down little by little as you keep on advancing through the book.
Still, couple of good learnings from it. Would recommend if you’re doing any kind of metrics related job and would like to get different perspectives.
Too many (really long) examples and quite repetitive take aways in my opinion, could be shorter.
Great research on every topic though.
Still, couple of good learnings from it. Would recommend if you’re doing any kind of metrics related job and would like to get different perspectives.
Too many (really long) examples and quite repetitive take aways in my opinion, could be shorter.
Great research on every topic though.
October 24, 2022
A good cautionary tale about using data for purposes it was not designed for. There are a good number of examples and the unforeseen impact when the data was used for more than intended. I did appreciate the summary in the last chapter of what to look for with bad data.
January 12, 2024
An interesting look at various ways we get measurements wrong. The book has some very good concepts that everyone involved with metrics and data should understand, but I'd say it's about 30% too long.
June 10, 2025
Overall a very good experience. The author explores his thesis using various pertinent and engaging examples and phenomenon, and never forgets the central point. The concluding chapters bring down the reading experience tho, as they feel repeatative. I would highly recommend this book.
January 20, 2021
This is a very good book not only for the ones that are working with index everyday but for all. The examples of bad data are astonishing!
September 4, 2022
Very interesting and gave me lots of ideas. Will apply the lessons learned to metrics I implement. Particularly enjoyed the stories and examples.
July 12, 2022
Reminds me quite a lot of "Proofiness: How You're Being Fooled by the Numbers" (which was also excellent).
Displaying 1 - 24 of 24 reviews