What do you think?
Rate this book
Uncontrolled by Jim Manzi
How do we know which social and economic policies work, which should be continued, and which should be changed? Jim Manzi argues that throughout history, various methods have been attempted—except for controlled experimentation. Experiments provide the feedback loop that allows us, in certain limited ways, to identify error in our beliefs as a first step to correcting them. Over the course of the first half of the twentieth century, scientists invented a methodology for executing controlled experiments to evaluate certain kinds of proposed social interventions. This technique goes by many names in different contexts (randomized control trials, randomized field experiments, clinical trials, etc.). Over the past ten to twenty years this has been increasingly deployed in a wide variety of contexts, but it remains the red-haired step child of modern social science. This is starting to change, and this change should be encouraged and accelerated, even though the staggering complexity of human society creates severe limits to what social science could be realistically expected to achieve. Randomized trials have shown, for example, that work requirements for welfare recipients have succeeded like nothing else in encouraging employment, that charter school vouchers have been successful in increasing educational attainment for underprivileged children, and that community policing has worked to reduce crime, but also that programs like Head Start and Job Corps, which might be politically attractive, fail to attain their intended objectives. Business leaders can also use experiments to test decisions in a controlled, low-risk environment before investing precious resources in large-scale changes – the philosophy behind Manzi’s own successful software company.In a powerful and masterfully-argued book, Manzi shows us how the methods of science can be applied to social and economic policy in order to ensure progress and prosperity.
Unknown Binding
First published May 1, 2012
33 people are currently reading
643 people want to read
Ratings & Reviews
Friends & Following
Create a free account to discover what your friends think of this book!
Community Reviews
Displaying 1 - 29 of 29 reviews
November 12, 2020
Moved to gwern.net.
March 2, 2017
The book jumps all of the place from philosophically-minded musings on foundations of science to business strategies, limitations of regression, history of social policy, nature of liberty and recommendations for the future. Unlike the fluctuations in the subject matter the writing itself is disappointingly constant in rhythm and tempo. Yes, it is clear and deliberate, but oh boy I wish Manzi experimented with his writing style as much as he advocates experimenting with everything else – the sections on business strategy and history of technical consulting were particularly mind-numbing. That said his tone was perfectly suitable for historiography of science while few sections of the book were genuinely intellectually stimulating, regardless of the form.
Notes:
- Nice description of how Bacon breaks with almost 2000 years of Aristotelian tradition of final causes and instead starts viewing science as a predictor rather than a truth finder. Instead of looking for the “essence” Bacon wants to start with physical facts and generalize from them. But not so fast - Hume throws in a little problem of induction that is quite upsetting to quite a few of well-meaning aspiring scientists. Luckily Popper sorta almost saves the day (not really) with falsification and Kuhn follows up with paradigm shifts and together they hold the hodge-podge of scientific method together. Messy as it is, it works out quite well for humanity.
- Cute parallels between science and markets where local self-centered behavior of individuals (dogmatism and self-interest respectively) gives rise to big benefits at higher level of abstraction (overall progress in science and benefits of market dynamics). Maybe, I buy it for markets, less so for science.
- Distinguishes between risk and uncertainty, very similar to Taleb’s theoretical casino probabilities vs randomness.
- Causal density is a big theme (low in physics, high in social science). The higher the density the more difficult to disentangle various potentially contributing causes.
- Talks about Comte and Saint Simone as godfathers of social science (and nobody trusted the French ever since ). JS Mill was hoping to generalize from history and then have future falsify the theory. Either way absence of controlled experiments can’t be overcome in social science to this day.
- Massive regression limitations in social sciences – omitted variable, correlation and higher-order interaction (change of direction of effect as a function of another variable). In regimes of high causal density interaction effect becomes dominant, no real way around it.
- RCTs is best we can do. At the limit views RCTs as an approximation of evolutionary process. Still hard to generalize from them, but at least they can demonstrate no effect, and replication crisis notwithstanding RCTs have remarkably better replication rates relative to no-RCTs which are totally hopeless.
- Status quo bias is often a rational preference for non-broken implicit knowledge and the onus should always be on the social policy change proponent so show effectiveness. RCTs can help weed out the noise, as vast majority of social interventions don’t work. Arnold Kling’s Null Hypothesis watch series is always a fun read…
- Some exceptions: social interventions that did work – “broken windows” policing and work requirement for welfare. Both introduced after extensive RCTs testing and seemed to have been verified by years of subsequent practice. O-oh don’t stop reading my liberal friend, there is some hope for you below.
- Manzi regrets inability of fed gov’t to systematically run the RCT experiments with the states, like it used to. In fact in the last part of the book among other things he suggests some version of federal agency explicitly dedicated to running and evaluating RCTs for social interventions.
- Distinguishes liberty-as-means vs liberty-as-goal which reminded me of David Friedman’s anti-deontological libertarianism. Liberty-as-means is freedom from absolute truth so you an experiment to see what works and if what works happens to violate a little bit of individual liberty then suck it up.
- Tension between innovation and social cohesion. Serious channeling of Tocqueville and Cowen here… You need innovation for growth and associated improvements in living standards across the board, but creating winners and losers tends to put pressure on social cohesion, thus need for some amount of gov’t-provided security (i.e. welfare state). But too much security and it will hamper innovation – disincentives for risk taking and complacency, erosion of the spirit. Still need the vessel of stable social fabric and rule of law to innovate and reap the fruits of it. Don’t over empathize “I did this alone” too much, communities are important. In the end right-off-center Manzi is kinda fumbling while trying to strike a balance between Obama’s “you didn’t build that” and John Gray’s notion that capitalism necessitates a strong state.
Notes:
- Nice description of how Bacon breaks with almost 2000 years of Aristotelian tradition of final causes and instead starts viewing science as a predictor rather than a truth finder. Instead of looking for the “essence” Bacon wants to start with physical facts and generalize from them. But not so fast - Hume throws in a little problem of induction that is quite upsetting to quite a few of well-meaning aspiring scientists. Luckily Popper sorta almost saves the day (not really) with falsification and Kuhn follows up with paradigm shifts and together they hold the hodge-podge of scientific method together. Messy as it is, it works out quite well for humanity.
- Cute parallels between science and markets where local self-centered behavior of individuals (dogmatism and self-interest respectively) gives rise to big benefits at higher level of abstraction (overall progress in science and benefits of market dynamics). Maybe, I buy it for markets, less so for science.
- Distinguishes between risk and uncertainty, very similar to Taleb’s theoretical casino probabilities vs randomness.
- Causal density is a big theme (low in physics, high in social science). The higher the density the more difficult to disentangle various potentially contributing causes.
- Talks about Comte and Saint Simone as godfathers of social science (and nobody trusted the French ever since ). JS Mill was hoping to generalize from history and then have future falsify the theory. Either way absence of controlled experiments can’t be overcome in social science to this day.
- Massive regression limitations in social sciences – omitted variable, correlation and higher-order interaction (change of direction of effect as a function of another variable). In regimes of high causal density interaction effect becomes dominant, no real way around it.
- RCTs is best we can do. At the limit views RCTs as an approximation of evolutionary process. Still hard to generalize from them, but at least they can demonstrate no effect, and replication crisis notwithstanding RCTs have remarkably better replication rates relative to no-RCTs which are totally hopeless.
- Status quo bias is often a rational preference for non-broken implicit knowledge and the onus should always be on the social policy change proponent so show effectiveness. RCTs can help weed out the noise, as vast majority of social interventions don’t work. Arnold Kling’s Null Hypothesis watch series is always a fun read…
- Some exceptions: social interventions that did work – “broken windows” policing and work requirement for welfare. Both introduced after extensive RCTs testing and seemed to have been verified by years of subsequent practice. O-oh don’t stop reading my liberal friend, there is some hope for you below.
- Manzi regrets inability of fed gov’t to systematically run the RCT experiments with the states, like it used to. In fact in the last part of the book among other things he suggests some version of federal agency explicitly dedicated to running and evaluating RCTs for social interventions.
- Distinguishes liberty-as-means vs liberty-as-goal which reminded me of David Friedman’s anti-deontological libertarianism. Liberty-as-means is freedom from absolute truth so you an experiment to see what works and if what works happens to violate a little bit of individual liberty then suck it up.
- Tension between innovation and social cohesion. Serious channeling of Tocqueville and Cowen here… You need innovation for growth and associated improvements in living standards across the board, but creating winners and losers tends to put pressure on social cohesion, thus need for some amount of gov’t-provided security (i.e. welfare state). But too much security and it will hamper innovation – disincentives for risk taking and complacency, erosion of the spirit. Still need the vessel of stable social fabric and rule of law to innovate and reap the fruits of it. Don’t over empathize “I did this alone” too much, communities are important. In the end right-off-center Manzi is kinda fumbling while trying to strike a balance between Obama’s “you didn’t build that” and John Gray’s notion that capitalism necessitates a strong state.
January 7, 2013
Manzi's book was of dual interest to me: I'd enjoyed his articles in magazines like National Review and at my last two companies I'd used and been very impressed by the Test & Learn software which the company he helped found (Applied Predictive Technologies) produces.
From the reviews I'd read, I expected the book to focus primarily on the potential for forming public policy based on test/control experiments. That is Manzi's object, but most of the book is actually focused on discussing the nature and development of the scientific method and of randomized field trials in particular as a means of developing empirical knowledge. This is the focus of the first two thirds of the book, and it is astoundingly well done. I would seriously consider using this portion of the book as part of a high school level science course because it does such a good job of discussing how the process of building up explanatory theories from observed and tested events works.
The last third of the book actually deals with the question of testing as a support for developing better public policy. This was interesting, but not as good as what came before. I do recommend the book overall, but I think it should be looked to more for a good summary of the epistemology of empiricism than as a policy book.
From the reviews I'd read, I expected the book to focus primarily on the potential for forming public policy based on test/control experiments. That is Manzi's object, but most of the book is actually focused on discussing the nature and development of the scientific method and of randomized field trials in particular as a means of developing empirical knowledge. This is the focus of the first two thirds of the book, and it is astoundingly well done. I would seriously consider using this portion of the book as part of a high school level science course because it does such a good job of discussing how the process of building up explanatory theories from observed and tested events works.
The last third of the book actually deals with the question of testing as a support for developing better public policy. This was interesting, but not as good as what came before. I do recommend the book overall, but I think it should be looked to more for a good summary of the epistemology of empiricism than as a policy book.
June 24, 2019
There is no way of knowing what works best in any given situation. No matter the outcome of an experiment, there is always the possibility that there are exceptions to the outcome. If the outcome shows an indication of a cause-and-effect relationship, it is possible that it was only applicable to a certain situational environment. If the outcome is a failure, it is possible that there are certain situations in which it can be true. This book asks for humility in our knowledge base, as what we know can be proven to be wrong given time.
For Manzi, the start of the scientific method was with Francis Bacon. Bacon made two observations which narrate the development of science. The observations include that the complexity of nature is greater than mental capacities of individuals to understand the complexity, and people find more patterns in data then there are. The scientific method changed the way in which the search for knowledge is carried out. Opponents of the scientific method are the Scholastic philosophers who sought knowledge from logic and debate without seeking new information or testing the theories.
The scientific method uses trial-and-error by holding as many potential causes constant and observing changes to the outcome by changing one potential cause. The evidence is inductive and can help with generalizations but with no certainty as to whether or not they are correct. Every generalization can have a variety of hidden conditionalities, creating the Problem of Induction. Generalizations are easier when there is low causal density such as biology. Unfortunately, social sciences have high causal density which means practical generalization are extremely rare. To find more of the conditionalities, experiments should be numerous and be varied.
Paradigms are priorly accepted generalizations. Paradigms are the status quo of assumptions. Scientists use paradigms to conduct theories and interpret data without the need to reconstruct every basic assumption. New information and observations which contradict the paradigm do not necessarily change the paradigm immediately. Over time, the paradigms do change and update to be inclusive of all available information. Practically, as Manzi puts it, paradigms are temporary systems of framework to aid in the construction of the new paradigms. New paradigms take over when the guardians of old paradigms retire or die, while new paradigms gain recruits. Paradigms need to be flexible to be able to progress knowledge while preventing endless questions of basic assumptions.
Trial-and-error is currently the best methods for the progression of knowledge and can be applied to private and public issues no matter the size. Making decisions about how society is to be organized should be embedded with incentives to experiment. Experimentation will lead any institution into continually searching for better organizational methods. The experimental knowledge obtained may not lead to immediate applications, but successive iterations of experimentation and practical applications interact with each other to produce further ideas.
There are two problems within the framework of this book which include limiting the history of knowledge and logic. The general assumption in this book is that knowledge will progress, while the method of progression is key. History shows that knowledge can be destroyed via destruction of libraries or certain paradigms removing reality from theories like the Scholastic philosopher. This brings up the second problem which is limiting the discussion on logic. The way the trial-and-error experiments are built are based on logic and theories. Manzi makes great effort to express the importance of Randomized Field Tests (RFTs), but their construction is based on logic. Trial-and-error can help filter information but it is not independent of logic.
The major lesson of this book is to not to presume that assumptions of knowledge are correct. Testing theories obtains necessary, but not sufficient, information to decide how current sets of decisions and institutions can be made better. As there are varied causal reasons for every generalization, no single test can prove or disprove knowledge. Manzi mantra to keep seekers of knowledge humble can be summarized as: “The model is never the system”.
For Manzi, the start of the scientific method was with Francis Bacon. Bacon made two observations which narrate the development of science. The observations include that the complexity of nature is greater than mental capacities of individuals to understand the complexity, and people find more patterns in data then there are. The scientific method changed the way in which the search for knowledge is carried out. Opponents of the scientific method are the Scholastic philosophers who sought knowledge from logic and debate without seeking new information or testing the theories.
The scientific method uses trial-and-error by holding as many potential causes constant and observing changes to the outcome by changing one potential cause. The evidence is inductive and can help with generalizations but with no certainty as to whether or not they are correct. Every generalization can have a variety of hidden conditionalities, creating the Problem of Induction. Generalizations are easier when there is low causal density such as biology. Unfortunately, social sciences have high causal density which means practical generalization are extremely rare. To find more of the conditionalities, experiments should be numerous and be varied.
Paradigms are priorly accepted generalizations. Paradigms are the status quo of assumptions. Scientists use paradigms to conduct theories and interpret data without the need to reconstruct every basic assumption. New information and observations which contradict the paradigm do not necessarily change the paradigm immediately. Over time, the paradigms do change and update to be inclusive of all available information. Practically, as Manzi puts it, paradigms are temporary systems of framework to aid in the construction of the new paradigms. New paradigms take over when the guardians of old paradigms retire or die, while new paradigms gain recruits. Paradigms need to be flexible to be able to progress knowledge while preventing endless questions of basic assumptions.
Trial-and-error is currently the best methods for the progression of knowledge and can be applied to private and public issues no matter the size. Making decisions about how society is to be organized should be embedded with incentives to experiment. Experimentation will lead any institution into continually searching for better organizational methods. The experimental knowledge obtained may not lead to immediate applications, but successive iterations of experimentation and practical applications interact with each other to produce further ideas.
There are two problems within the framework of this book which include limiting the history of knowledge and logic. The general assumption in this book is that knowledge will progress, while the method of progression is key. History shows that knowledge can be destroyed via destruction of libraries or certain paradigms removing reality from theories like the Scholastic philosopher. This brings up the second problem which is limiting the discussion on logic. The way the trial-and-error experiments are built are based on logic and theories. Manzi makes great effort to express the importance of Randomized Field Tests (RFTs), but their construction is based on logic. Trial-and-error can help filter information but it is not independent of logic.
The major lesson of this book is to not to presume that assumptions of knowledge are correct. Testing theories obtains necessary, but not sufficient, information to decide how current sets of decisions and institutions can be made better. As there are varied causal reasons for every generalization, no single test can prove or disprove knowledge. Manzi mantra to keep seekers of knowledge humble can be summarized as: “The model is never the system”.
May 15, 2021
[Imported automatically from my blog. Some formatting there may not have translated here.]
The book's subtitle is "The Surprising Payoff of Trial-and-Error for Business, Politics, and Society." To get there, Manzi starts in a surprising place: the philosophy of the scientific method, going back to folks like Aristotle and Francis Bacon.
Which makes sense, sort of. While the scientific method has successfully managed to grow our knowledge of the physical world, its applicability to figure things out in other fields is problematic. Determining how long to fire a rocket engine in order to have your space probe hit the Martian atmosphere at precisely the right location and angle—that's simple in comparison with (say) figuring out how to price thousands of items in your store in order to maximize revenue.
Manzi argues that it's possible to "do science" in matters of what he calls "high causal density": specifically, cases involving the behavior of large numbers of people. While the classic "controlled experiments" of science don't translate well to this arena, the technique of "randomized field trials" shows more promise. It's a technique already widely used in business and (some) social sciences. Manzi urges its wider adoption in political issues as well.
At the end of the book, Manzi discusses how his insights might apply to education, immigration, and welfare. This bit will appeal to political types who want policies that "work". (As opposed to political types who say they want policies that work, which just happen to be the policies demanded by their political party.)
January 9, 2023
"Dropping balls off a tower allows us to overrule Aristotle."
A really interesting read starting with a very brief history of the philosophy of science, arguments for why randomised experimentation is the best tool we have to acquire new knowledge, then giving a review of the state of experimentation in various sectors.
This is a humbling book for everyone.
For those without experimentation backgrounds, it's a humbling read about the limits and fragility of practical knowledge fullstop, but particularly for ideas that have not undergone experiments.
For those with an experimentation background, it's a reminder that, simply put, most experiments won't work out. Any system today is a result of some combination of trial-and-error and copying of policies that participants have found useful elsewhere or observed in other organisations. It can therefore be quite hard to come up with new ideas that will have an impact. But we still must try to iterate on what we have and learn along the way. An additional humbling point for experimenters is that ideas which would have an enormous positive impact rarely exist- in part because someone will have found them already and in part because unlike physics, most real world problems are 'divergent' in nature- i.e. they have multiple underlying causes and each cause has its own causes and so on.
A really interesting read starting with a very brief history of the philosophy of science, arguments for why randomised experimentation is the best tool we have to acquire new knowledge, then giving a review of the state of experimentation in various sectors.
This is a humbling book for everyone.
For those without experimentation backgrounds, it's a humbling read about the limits and fragility of practical knowledge fullstop, but particularly for ideas that have not undergone experiments.
For those with an experimentation background, it's a reminder that, simply put, most experiments won't work out. Any system today is a result of some combination of trial-and-error and copying of policies that participants have found useful elsewhere or observed in other organisations. It can therefore be quite hard to come up with new ideas that will have an impact. But we still must try to iterate on what we have and learn along the way. An additional humbling point for experimenters is that ideas which would have an enormous positive impact rarely exist- in part because someone will have found them already and in part because unlike physics, most real world problems are 'divergent' in nature- i.e. they have multiple underlying causes and each cause has its own causes and so on.
December 2, 2022
A fairly sober and objective take on business, political, and social improvement through the lens of scientific thinking. Breaks down the history of science and experimentation itself and then how it has become more integrated into all types of organizations.
Always provides rationale for opinions and clearly lays out the tradeoffs inherent to many methods of science. Strong endorsement of pushing experimentation and open-minded learning through all sectors/institutions in the modern world and the potential benefits it can bring us if we decide to pursue evidence and sense-making collectively as a whole.
The book ends in an epic quote that sums up the argument nicely: "Each society, just as each individual, must find a way to compete in this world without losing its soul."
Always provides rationale for opinions and clearly lays out the tradeoffs inherent to many methods of science. Strong endorsement of pushing experimentation and open-minded learning through all sectors/institutions in the modern world and the potential benefits it can bring us if we decide to pursue evidence and sense-making collectively as a whole.
The book ends in an epic quote that sums up the argument nicely: "Each society, just as each individual, must find a way to compete in this world without losing its soul."
August 20, 2018
Very cool context for the whole undertaking of empiricism in applied social science over the broad sweep of civilized history. Starts very theoretical and then gets to some concrete applications with real problems today. Lags at points and it's not the most riveting read, but it's certainly clear and brings together the conversation that has happened around empiricism in social science in a good way. Very good read for anyone who is serious about bringing evidence to business or public policy.
June 5, 2017
Much of my professional life leans heavily on understanding the signal through the noise, and I'm a huge fan of the role of randomized controlled trials (or, per Manzi, field trials) in contemporary public policy. This does a good job. Five stars for content, three for structure and story-telling.
May 19, 2021
Terrific read covering the origins of the scientific method to how we can apply that methodology to the social sciences, why it is important to do so, and why it has been and will continue to be a challenging implementation
April 25, 2023
Manzi goes above and beyond in this book, bringing in a surprising amount of philosophy of science and statistical rigor. As far as business books go, you won't find a more insightful "academic" book in the popular press.
August 4, 2024
Honestly didn’t enjoy this book. Took way too long to get to any point and was less enjoyable to read than your average college textbook
January 9, 2019
Great overview of philosophy of science, then proceeds to link the philosophy with genetic algorithms(probably has one of the best discussions on those i've seen). The meat of the book is about how businesses generate knowledge or how they are actually much closer to scientific than one thinks generally thinks they are and finally there are some policy recommendations using the principles developed. I mostly enjoy this book for its exposition chapters, I found some of the discussions a bit repetitive but its a book I would recommend nonetheless.
July 18, 2012
Uncontrolled: The Surprising Payoff of Trial and Error for Business, Politics and Society by Jim Manzi is an excellent book that looks at the use of randomized field trials (RFTs) for public policy, business and society. The book advocates that RFTs be used as much as possible.
The book is divided into three parts, in the first Manzi looks at Science and how it changed to use empirical validation, in the second Social Science is examined and finally in the third Political Action is where Manzi makes recommendations.
In the first section on Science Manzi looks at the origin of the scientific method and implicit and explicit knowledge. Implicit knowledge is defined by Manzi as the knowledge within a system of what works reasonably well even if it is not conscious. He uses genetic algorithms as an example of this kind of knowledge being present in a system. Manzi goes on to suggest that Scientific advancement is similar to evolution in that theories that can be tested and that are proven right survive. He provides the failure of scientific advancement without empirical validation as being important. Finally he goes on to describe RFTs as they are used in science, particularly in medical trials.
In the second part of the book Manzi talks about his experience as a consultant and how consultants constructed models that were not greatly useful but would up looking at RFTs that did provide utility. He also talks about the problems with such trials and how they need a lot of good data and are sometimes not as good as the wisdom of experts. He talks about how RFTs have been made to work in business and provides the example of the Capital One credit card company in the US.
Then he looks at how these trials can be used with crime, welfare, education, economics and political science. He also shows that some of the scientific studies that are widely quoted, such as the study that showed that an over abundance of choice reduced consumption were flawed. Manzi emphasizes that iterative trials should be used to avoid such problems.
The final part of the book on Political Action is not particularly strong. Manzi does provide an interesting look at what he terms the paradox of Liberty and ponders how far liberty should be taken. Finally he provides some recommendations that are to use RFTs more, which is fairly sensible but he also provides some far weaker suggestions of his own including the well worn suggestion to spend more or science and technology which appears to come more from Manzi's background as a math major at MIT than from the rest of the book.
The strength of the book in the first two sections is terrific though. It is a terrific, fairly short and well written public policy book. Anyone who works in public policy would be well served by reading it. For anyone interested in science and social science it's also very much worth reading.
The book is divided into three parts, in the first Manzi looks at Science and how it changed to use empirical validation, in the second Social Science is examined and finally in the third Political Action is where Manzi makes recommendations.
In the first section on Science Manzi looks at the origin of the scientific method and implicit and explicit knowledge. Implicit knowledge is defined by Manzi as the knowledge within a system of what works reasonably well even if it is not conscious. He uses genetic algorithms as an example of this kind of knowledge being present in a system. Manzi goes on to suggest that Scientific advancement is similar to evolution in that theories that can be tested and that are proven right survive. He provides the failure of scientific advancement without empirical validation as being important. Finally he goes on to describe RFTs as they are used in science, particularly in medical trials.
In the second part of the book Manzi talks about his experience as a consultant and how consultants constructed models that were not greatly useful but would up looking at RFTs that did provide utility. He also talks about the problems with such trials and how they need a lot of good data and are sometimes not as good as the wisdom of experts. He talks about how RFTs have been made to work in business and provides the example of the Capital One credit card company in the US.
Then he looks at how these trials can be used with crime, welfare, education, economics and political science. He also shows that some of the scientific studies that are widely quoted, such as the study that showed that an over abundance of choice reduced consumption were flawed. Manzi emphasizes that iterative trials should be used to avoid such problems.
The final part of the book on Political Action is not particularly strong. Manzi does provide an interesting look at what he terms the paradox of Liberty and ponders how far liberty should be taken. Finally he provides some recommendations that are to use RFTs more, which is fairly sensible but he also provides some far weaker suggestions of his own including the well worn suggestion to spend more or science and technology which appears to come more from Manzi's background as a math major at MIT than from the rest of the book.
The strength of the book in the first two sections is terrific though. It is a terrific, fairly short and well written public policy book. Anyone who works in public policy would be well served by reading it. For anyone interested in science and social science it's also very much worth reading.
This entire review has been hidden because of spoilers.
June 7, 2014
In a sense, I think this is the best book that Nassim Taleb never wrote. Like Taleb's books, it has a central theme but is at its heart a grab bag of ideas and musings on a variety of topics trimmed heroically by the editors. It explores many of Taleb's favorite themes, like uncertainty and the fallibility of technology, but the treatment is more disciplined and has none of Taleb's annoying indulgences.
As others reviewers have pointed out, Manzi often demonstrates an admirable restraint and epistemic humility in his writings. I think he is very successful in explaining the limits of mathematics and technology and demonstrating the need for systematic randomized field trials (this is really what the title means by "trial-and-error") to stake out improvements. His long experience as a practitioner of quantitative business analysis gives these arguments a lot of extra weight. To my general state of technological optimism and data-driven exuberance, I found this quite sobering and it made me uncertain of how "limitless" the possibilities of data-related technologies really are, especially when applied to areas of high causal density like finance, economics and social policy.
I was not aware of Manzi's background as a National Review contributor prior to reading the book, but knowing this after the fact made a lot of inclusions make more sense. For example, he would refer casually to the "libertarian" position without introducing it properly, and addresses a number of hot-button topics in US politics like school vouchers and welfare reform in a way that assumes the reader to be familiar with them, and sympathetic to the positions that an American conservative would usually hold. This would make sense if he was writing to his audience of NR readers, but since this book is presumably meant for a general audience, it came across rather odd.
Unfortunately, as the book becomes more political, it loses the intellectual rigor that makes its beginning sections on the development of the scientific method so appealing. This is especially evident in the last section, where he says he will make a set of policy recommendations based on the book's central theme of trial-and-error, but really ends up writing a National Review column providing provisional support to most of the solutions favored by Buckley-style intellectual conservatives. Shadows of moderates like Douthat and Brooks, both of which are thanked at the end. If that is your thing, good for you, but not what I signed up for at all.
Still, the beginning parts are excellent and the book serves well as a useful antidote to the current hyperbole surrounding advanced analysis methods. Three stars and a recommendation from me.
As others reviewers have pointed out, Manzi often demonstrates an admirable restraint and epistemic humility in his writings. I think he is very successful in explaining the limits of mathematics and technology and demonstrating the need for systematic randomized field trials (this is really what the title means by "trial-and-error") to stake out improvements. His long experience as a practitioner of quantitative business analysis gives these arguments a lot of extra weight. To my general state of technological optimism and data-driven exuberance, I found this quite sobering and it made me uncertain of how "limitless" the possibilities of data-related technologies really are, especially when applied to areas of high causal density like finance, economics and social policy.
I was not aware of Manzi's background as a National Review contributor prior to reading the book, but knowing this after the fact made a lot of inclusions make more sense. For example, he would refer casually to the "libertarian" position without introducing it properly, and addresses a number of hot-button topics in US politics like school vouchers and welfare reform in a way that assumes the reader to be familiar with them, and sympathetic to the positions that an American conservative would usually hold. This would make sense if he was writing to his audience of NR readers, but since this book is presumably meant for a general audience, it came across rather odd.
Unfortunately, as the book becomes more political, it loses the intellectual rigor that makes its beginning sections on the development of the scientific method so appealing. This is especially evident in the last section, where he says he will make a set of policy recommendations based on the book's central theme of trial-and-error, but really ends up writing a National Review column providing provisional support to most of the solutions favored by Buckley-style intellectual conservatives. Shadows of moderates like Douthat and Brooks, both of which are thanked at the end. If that is your thing, good for you, but not what I signed up for at all.
Still, the beginning parts are excellent and the book serves well as a useful antidote to the current hyperbole surrounding advanced analysis methods. Three stars and a recommendation from me.
October 5, 2015
If you approach this book as a beginning to further inquiry, you're going to pleased. Manzi does an admirable job of reviewing the philosophy of science (for non specialists, and in VERY broad strokes), and in describing the development of randomized field trials in medicine, both of which form the basis for his subsequent discussion of the value of experimental approaches to strategy in public policy and business. This discussion is not a how-to. You will be disappointed if you want a primer on developing and executing strategic experiments. Also missing is any discussion of the risks of experimenting on customers (that's right, Facebook, I'm looking at you...). But the book asks the right questions, and one good question is worth several mediocre explanations. I admire Manzi's work. He and others like him are onto something. Unfortunately, most managers and government officials don't understand basic scientific reasoning, nor do they grasp the core principles of experimentation. This book is a great start in redressing that gap.
December 26, 2013
No surprise. A CEO of a company that designs controlled experiments thinks that controlled experiments are a good idea.
With caveats
1. Controlled experiments are a necessary but not sufficient condition for scientific progress.
2. Replicating experiments is required before drawing conclusions.
3. Experimentation does not occur and results may not be used in a vacuum.
With skepticism
1. New programs seldom work.
2. Programs that attempt to raise skills or consciousness are more likely to fail than those which change incentives and environment.
3. There is no magic
Finally
High casual densities mean that every result will be conditional
Many core questions in social science are so holistically integrated that experiments are impossible
With caveats
1. Controlled experiments are a necessary but not sufficient condition for scientific progress.
2. Replicating experiments is required before drawing conclusions.
3. Experimentation does not occur and results may not be used in a vacuum.
With skepticism
1. New programs seldom work.
2. Programs that attempt to raise skills or consciousness are more likely to fail than those which change incentives and environment.
3. There is no magic
Finally
High casual densities mean that every result will be conditional
Many core questions in social science are so holistically integrated that experiments are impossible
Read
June 20, 2013'Human beings crave certainty.
Throughout history, assorted shamans, haruspices, auspices, astrologers, sibyls, kaballahists, pyromancers, Hegelians, Marxists, palmists, tarot-card readers, stock chartists, and computer modelers have made good livings off of the apparently limitless market demand for more certainty and reduced risk.
But as Jim Manzi persuasively argues in his insightful and well-written new book, Uncontrolled, humanity is terrible at foresight, and trial-and-error is the chief way humans develop reliable knowledge.'
Read the full review, "The Science of Policy," on our website:
http://www.theamericanconservative.co...
Throughout history, assorted shamans, haruspices, auspices, astrologers, sibyls, kaballahists, pyromancers, Hegelians, Marxists, palmists, tarot-card readers, stock chartists, and computer modelers have made good livings off of the apparently limitless market demand for more certainty and reduced risk.
But as Jim Manzi persuasively argues in his insightful and well-written new book, Uncontrolled, humanity is terrible at foresight, and trial-and-error is the chief way humans develop reliable knowledge.'
Read the full review, "The Science of Policy," on our website:
http://www.theamericanconservative.co...
August 18, 2012
This book is a hodgepodge of different things: part history of science, part reflection on political theory, part memoir, and part policy tract. Still, even with all of these disparate elements, Manzi managed to combine relentless logic with epistemic humility throughout this book. (I see this as a model for clear thinking, and it's one I try to mimic, usually without complete success.) He is probably the best thinker on the policy-oriented right, and this is the most thoughtful, interesting book I've read all year.
July 12, 2013
1 - There are very strong limits to what we can know about the causes and consequences of human intervention in business, public policy, etc
2 - Randomized Field Trials are the best of bad options at delivering useful knowledge, and they ought to be employed widely, with the understanding that they will usually find no effect and almost never find large effects.
3 - Public policy should reflect an experimental ethos - waivers for states to try new things, school choice, immigration to attract scientists and engineers, etc
2 - Randomized Field Trials are the best of bad options at delivering useful knowledge, and they ought to be employed widely, with the understanding that they will usually find no effect and almost never find large effects.
3 - Public policy should reflect an experimental ethos - waivers for states to try new things, school choice, immigration to attract scientists and engineers, etc
January 28, 2015
I work for Manzi's company, so I have absolutely drank the testing kool aid and believe in the arguments made in this book. I enjoyed the many detailed examples and appreciated the scientific history presented in the first few chapters, but the book got wordy at places and tended to repeat itself a lot. I think the same material could have been presented more effectively in half the length. If you're interested in the subject but want the sparknotes version, see Manzi's article in the December 2014 issue of the Harvard Business Review.
October 5, 2014
Impressed with the history of science and thorough discussion of Popper and falsificationism, but disappointed in the level of detail given to experimentation in contemporary public policy. And, minor annoyance, why does Manzi refer to experiments as RFTs instead of RCTs or RTs?
April 7, 2013
This book, alas, needed to be shorter. The author padded out a bunch of insights with a history of science and some vague policy recommendations. The good parts, on the practical use of experiments in business and government, were very good.
April 5, 2015
I really enjoyed this book. It was about Randomized Field Trials and their many applications. It also had a good bit of myth busting. I especially liked the criticism of the Freakonomics case for abortions being the reason for the reduction in crime.
June 6, 2012
Worth a read. Manzi knows a ton about the subject matter, having been in the business analytics sector for a while. You'll appreciate just how HARD social science really is at the end of it.
June 9, 2012
Good solid stuff on philosophy of science, the virtues of experimentation, etc., from one of the (technocratic) (center-)right's best thinkers.
January 25, 2013
A strange mash-up of statistics, self-aggrandisement, and boilerplate libertarian policy recs.
Read
March 29, 2014Great introduction into the value of a scientific method to inform social policy decisions.
April 28, 2017
I FINALLY FINISHED!!! I thought it was so intelligent and thoughtful throughout, I just had difficulty pushing through because it's more serious than most stuff I read. I renewed it more times than I thought the library system would let me. tl;dr we can't know things with certainty unless we run good experiments. The less rigorous the experiment--however plausible a setup--the less confidence we should have in the conclusions.
Anyway, if you consider yourself someone who thinks we (in business or in politics) should be making more science- and evidence-based choices, this is a must-read to have credibility in that position.
Since most people who identify along those lines would likely also generally consider themselves progressives or liberals, I feel I should note that the author seems to me to be pretty clearly on the libertarian/conservative end of things because of the epistemological humility that actually using the scientific method requires. Aka being "For Science" is a lot more difficult than, "we should enact the policy preferences of the academics and experts that we can imagine being friends with and have the same moral and social values that we do," because there really is very little that we actually understand all that well. "Everyone knows that..." (and the implied, "people who disagree with me are just ignorant/fearful/racist/sexist) is used far, far more often than is deserved.
Anyway, if you consider yourself someone who thinks we (in business or in politics) should be making more science- and evidence-based choices, this is a must-read to have credibility in that position.
Since most people who identify along those lines would likely also generally consider themselves progressives or liberals, I feel I should note that the author seems to me to be pretty clearly on the libertarian/conservative end of things because of the epistemological humility that actually using the scientific method requires. Aka being "For Science" is a lot more difficult than, "we should enact the policy preferences of the academics and experts that we can imagine being friends with and have the same moral and social values that we do," because there really is very little that we actually understand all that well. "Everyone knows that..." (and the implied, "people who disagree with me are just ignorant/fearful/racist/sexist) is used far, far more often than is deserved.
Displaying 1 - 29 of 29 reviews