The Hidden Half: How the World Conceals its Secrets

by Michael Blastland

Why does one smoker die of lung cancer but another live to 100? The answer is 'The Hidden Half' - those random, unknowable variables that mess up our attempts to comprehend the world.We humans are very clever creatures - but we're idiots about how clever we really are. In this entertaining and ingenious book, Blastland reveals how in our quest to make the world more understandable, we lose sight of how unexplainable it often is. The result - from GDP figures to medicine - is that experts know a lot less than they think. Filled with compelling stories from economics, genetics, business, and science, The Hidden Half is a warning that an explanation which works in one arena may not work in another. Entertaining and provocative, it will change how you view the world.

Genres: Science, Nonfiction, Psychology, Economics, Philosophy, Sociology, Mathematics

307 pages, ebook

First published April 4, 2019

Reviews

[Trevor · Rating 5 out of 5]

When I got about halfway through this – yes, I know, I do get irony – I had contacted about 5 friends and told them they needed to read this. With a few of them I’d discussed or even written papers on education and complexity theory. The thing is that I generally used ‘the bike and the hamster’ as an explanation of the difference between the complicated and the complex. You know, a bike is complicated – it has lots of bits that need to fit together in very specific ways or it won’t work. But you can take it apart and oil it and put it back together and you can even leave certain parts off – the mud guards or the bell – and it will work, perhaps even better than before.

You can’t do that with a hamster. Rip its leg off and it really won’t thank you. And even if you stick its leg back on really fast, it still won’t thank you. If it doesn’t die, it might end up in so much shock as to make it little different to being dead for whatever purpose you were using it for previously. The bike is complicated, the hamster is complex. Complex things don’t really do the ‘all else being equal’ thing. Running the same experiment multiple times to see if you can get the same result is possible with complicated things, but much less so with complex systems. That’s the point, by the way, complex things tend to be in systems and so the flap of butterfly wings can have insanely large impacts over time. And since we can’t input all of the wings of butterflies into our models, our models can only take us so far.

How far? Well, that’s the thing, isn’t it? This book doesn’t argue that we can know nothing at all – that would be clearly absurd – what it does argue, though, is that we tend to exaggerate how much we can and do know, and that’s where we get into trouble.

My favourite bit of this book is chapter 1. If you can just nip into a bookstore and tuck yourself in a corner and just read that, you really will be doing yourself a favour. In part this chapter helps explain why the book ended up being called the hidden half. It’s because mostly our best explanations for things only really explain about half of the reasons things happen. The example given is cancer. You may not know this, but breast cancer is a thing – I know, crazy – but while explaining why George got cancer and Gary didn’t can start lots of arguments from those who particularly like genetic explanations and those who like environmental explanations, these are harder to have when a woman has cancer in one breast and not the other.

So, you can say Gary developed cancer because he smoked too much, or because his grandfather had the same cancer, but it’s hard to say Sally’s right breast was particularly promiscuous or had worse genes that her left breast. Years ago a friend of mine told me of research that said women tend to have less cancer in the breast that their husband fondles the most. Far be it from me to cast doubt on such a theory – all the same, if I was a betting man, I wouldn’t mind putting money on the fact the scientist that discovered this correlation was a man. I could be wrong, admittedly, but breast cancer is still a thing and breast fondling stations haven’t become something you see at pink-ribbon events – ‘Nah, love, this is purely for your own good…’

The thing is since Sally’s left breast has more or less the same genes and has experienced more or less the same environment as Sally’s right breast, why did she get cancer in one and not the other? The answer is that we don’t really know. It seems that there might well be millions of factors (those damn butterflies again) that might contribute to whatever triggers cancer. We like big explanations – her left breast liked to smoke, or her grandmother on her father’s side had breast cancer in one breast too – but actually, often the big explanations are swamped by the tiny and even imperceptible differences that occur that are likely to be below our awareness, but that, over time (or perhaps all at once) mean one breast is fine and the other one not.

I’m really not trying to make light of breast cancer – the point is that our big explanations of genetics or environment seem to be stumped by the idea we end up with cancer in one and not the other. This is also true of testicles, of course, and arms, and legs, and eyes, I guess too.

There is a nice bit nearer the end of this where he explains some of the problems with statistics (jump to Chapter 6 if you’re in the bookshop and have a bit more time to flick through). You know, we stop doing things when we are told that they produce a 20% increase in your likelihood to die from something. But while that sounds dreadful, it might not be nearly as bad as you might initially think. So, let’s say taking a particular tablet increases your likelihood of dying from sunstroke than people in the population generally. That sounds pretty bad. But what proportion of the population dies from sunstroke in any given year, or even in a lifetime? One in a million? One in a hundred thousand? Probably not even… So, a twenty percent increase in likelihood of dying from sunstroke doesn’t even bring your chances up to two in a million.

There’s lot of interesting information like this along the way – if you’ve read Chapter 1 and 6 and the kids aren’t screaming that they want to go home, maybe keep reading through Chapter 7 too. We like to think that if something is big and shocking it must have lots to say to us about the way the world works. As he says, it is over a decade since the global financial crisis and all of the crazy left wing guys like me are saying it proves we need more regulation in the market and all the totally stupid right wing free market madmen are saying it proves we need much less regulation in the market. The thing is that big complex phenomena leave enough ambiguity for favourite theories to sneak through unharmed.

This book is well worth getting your hands on. I’m going to admit that it wasn’t quite as good at the end as it was at the start – but it was off the scale good at the start. So good it became urgent that I recommend it to my friends. I don’t really do that very often.

[Daniel · Rating 5 out of 5]

Causality is really, really hard and many solutions do not work. Why?

1. When genetically identical Marmorkreb crayfish are grown in the same environment and given lots of food, they all grow to very different sizes (published in Nature!), it proves that we simply don’t know what affects their size on top of their gene and environment. It is not bias or measurement error, but a fundamental property.
2. The biggest study about delinquent boys and adult crime found absolutely no factors that can predict later crime. Sometimes a date or a wife’s advice changed destiny. Conjoint twins sharing the same gene and environment have vastly different characters because of ‘Context specific learning’.
3. The above is complicated by the lack of consistency in us. Read Nick Chater’s The Mind is Flat for details. ‘Birds of the same feather flock together, but opposites attract.’
4. What works in one place may not apply to another, because of the weakest link problem. GDP numbers are notoriously prone to many revisions, sometimes by large amounts! Composition of economies change over time, method of measurement change, sampling changes from chance, lessons learnt during one period do not apply to the next. And then there is Knightian (radical) uncertainty, the unknowable unknowns.
5. Glenn Begley, Amgen’s ex-head of a research group at Amgen, found 90% of scientific studies not reproducible. Positive results are published much more than negative ones, and then there is “researchers’ degree of freedom”, that is, the many ways that a researcher can look at data.
6. Don’t believe in one magic button or theory that will solve intractable problems, such as poverty. Workers on the field need to get to know the aid recipients, find out their beliefs to really really help them.
7. What works for big population like stopping smoking in reducing cancer may not work for the individual. Sometimes there is no one main cause of something, such as drop of teenage pregnancy across developed countries. Big successes require a lot of things to be right and this may not be repeated even by the same person.
8. David Autor: On average, trade with China benefited everyone. In particular, lots of low wage white workers with less education were severely affected permanently. They lost their jobs and could not find another. So Donald Trump happened.
9. So what to do? Try:

1. Experiment and adapt. John Kay: disciplined pluralism; try many things, give up those that don’t work.
2. Triangulate, prove something with more than one method
3. Get some negative capability; avoid the tendency to conclude early
4. Embrace uncertainty; sometimes that is a virtue
5. Remember we are always betting
6. Communicate the uncertainty;
7. Govern for uncertainty
8. Manage for uncertainty
9. Don’t use probability to hide ignorance
10. Change your metaphors; remember many things can go wrong
11. Treasure your exceptions; they disprove the general
12. Relax

One can feel disheartened by this book. We all strive to learn and improve our lot; this book tells us that sometimes reasons are unknowable and interventions often fail. However it is very realistic; or else we will be shocked when well intentioned programs which were carefully thought through fail spectacularly. We need to be much more humble in our ability to understand, and to intervene.

In real life, all the planning will not last half a minute into the actual doing. All entrepreneurs understand this. That’s why those who succeed are people who try, fail, improve and try again. Some never succeed, and some do. But for those who are too pessimistic to try, nothing will ever happen.

Do circumstances make the hero, or the hero create circumstances for success? The Chinese believe that both are correct and the 2 phenomenon are not mutually exclusive. And Louis Pasteur said, ‘Chance favours the prepared mind.’

A solid 5 star book!

[Brian Clegg · Rating 5 out of 5]

Michael Blastland is co-author of one of my favourite titles on the use and misuse of numbers, The Tiger that Isn't - so I was excited to see this book and wasn't disappointed.

Blastland opens with the story of a parthenogenic crustacean that seems to demonstrate that, despite having near-identical nature and nurture, a collection of the animals vary hugely in size, length of life and practically every other measure. This is used to introduce us to the idea that our science deals effectively with the easy bit, the 'half' that is accessible, but that in many circumstances there is a hidden half that comprises a whole range of very small factors which collectively can have a huge impact, but which are pretty much impossible to predict or account for. (I put 'half' in inverted commas as it might be fairer to say 'part' - there's no suggestion that this is exactly 50:50.)

We go on to discover this hidden half turning up in all kinds of applications of science and maths from economics to healthcare, from the lives of individuals who were delinquent as children to measuring the benefit (or not) of giving a poor family a cow. Blastland picks out a number of factors that tend to mislead us - that as humans we aren't consistent in our responses, that things change with location and time, that the way research is undertaken can generate false results, that principles can mislead us and far more. As he points out, what this does is make us aware of the limitations of applied maths (particularly economics) and science. He is not saying we shouldn't use them - they are far better than the alternatives - but we have to be aware of their limitations.

At the end of the book, Blastland comes up with some suggestions for dealing with the hidden half. It isn't going to go away, and there are no easy solutions, but he does have some interesting ideas on mitigation. For example, he suggests we should experiment more (with political policies, for example), remember that we are betting on knowledge rather than making use of certainty and, for me, most importantly that scientists, journalists and politicians should do more to communicate the uncertainty involved. It's not that we want our politicians to be hesitant, but it would be far better if they made it clear that there are very few clear linkages between policies and outcomes. I was particularly struck by some data on GDP figures. The media and politicians often spend ages agonising over a GDP change of, say 0.2 per cent. The headline figures are revised over time as better data is available: Blastland points out that a typical correction might be 0.4 per cent up or down (and can be as high as 1 per cent). This makes it very clear that making pronouncements based on a 0.2 per cent change is futile and highly misleading.

My only criticism is that I felt that the book could have done with even more specific examples: in this kind of book, it's the examples that have the real bite and savour. Blastland spends a bit too long philosophising on the hidden half in a way that feels a little repetitive. It's not there aren't great examples (plenty more than the ones mentioned above), it was just for me that the ratio of examples to musings wasn't quite right. A brief mention also to the cover design - it's very clever (though I couldn't help thinking it had been put on wrong and wanting to re-position it).

We are getting more books now about the reality of applied maths (particularly economics) and science, which is an extremely good thing, provided the message is carried through to journalists and others who have to communicate these matters to the public. An excellent book.

[Michael Hudson · Rating 2 out of 5]

In the quippy social media vernacular: This book could have been an email.

Okay, that's the disappointment of a reader who had high expectations talking. The truth is, this book would be an excellent long article, or long form podcast. In fact, it was the latter. I found the book via a long form interview with the author on the EconTalk podcast and I was genuinely excited to read it. I still highly recommend that podcast episode, particularly if you intend to give the book a pass.

Unfortunately, the best bits of the movie were in the trailer. Sure, there were some real gems to take away from the book and I'm never sorry to receive a reminder to hold on to epistemic humility, but in the end the central point was over-proved and once that had been done the rest felt like filler.

To be fair to the author, parts of that filler were ... I'll use the word charming, and he is not unskilled at his craft. I'm quite certain this is some reader's five star book, it's simply not mine.

[David Wineberg · Rating 4 out of 5]

Michael Blastland is upset that everything is not neat and tidy. That there is no guarantee of symmetry. That economies are not predictable. That genetics does not describe nearly everything about life. In his The Hidden Half, he examines a multitude of disciplines and events to show we must be missing half of what is going on, because we can’t explain them otherwise. We exist on half the knowledge we need, without knowing what we don’t know.

The book is a fast-reading and delightful collection of failures, peppered with behavioral science, which is always entertaining.

When people who have written down a position one way are interviewed as if they had chosen the opposite stance, they actively defend the position they did not take as if it were always their own. Politicians are famous for being absolutely certain of the rightness of their position one day, yet take the precise opposite position another. Every year, only one economist is correct about the performance of the economy. Every model is wrong. Nothing, it seems, is predictable.

He begins with a startling enigma, at least to scientists. The marmorkrebs is a newly discovered kind of crawfish that flourishes in the German aquarium industry. Its unique feature is that the females can lay eggs that will hatch without input from males (parthenogenesis). All you need is a female, who can lay thousands of eggs and produce thousands of offspring with DNA identical to hers. Perfect clones, in theory. And yet, the offspring come in all shapes and sizes, and variable colorings. How is this possible when their DNA is identical? Blastland says nobody knows. No answer satisfies, and it all points out that we clearly do not understand genetics after all. If more proof were needed, he points to the past decade of breakthroughs in genetic research, which have resulted in essentially nothing. Identifying genes has not changed medicine or lives anything like the predictions had it, because that is only part of the story. And we don’t know what the rest of the story is.

Man has an insatiable need to put everything in its proper place. He needs to know things are organized, measurable, consistent and predictable. And they just aren’t. But that stops no discipline working on those assumptions and making those kinds of claims. Studies in peer-reviewed journals attest to the constant flow of new, absolutely proven ideas that are just plain wrong. Some cannot be replicated. Many are just the survivors; the journals don’t publish all the contradictory failure articles. So they aren’t cited in other papers. The result is undisputed discoveries that are worthless. We see them daily, particularly in biology, genetics and medicine. As easy to disprove as they were to prove, they are soon forgotten when they prove useless.

Medicine comes in for a particular beating in The Hidden Half.

Blastland deconstructs studies to show how useless they, drugs and tests really are. He shows that 90% of drug study results are not replicable, even by the original researchers. In one specific example of pointless tests, he takes on dementia in over 65s, where a test long considered reliable can pinpoint four positive cases in 100 tests. Unfortunately, there are six cases per hundred, so it misses two of them, or one third. Far worse, it also labels 23 additional cases as positive – falsely. This means the test claims a total of 27 positives when we know there are only six. The result: nearly two dozen people suffer the stress and anxiety of becoming demented without every becoming demented. So with breast cancer and numerous other examples where failure stops no one from taking these tests. Or doctors from requiring them.

Even in detective work, we have no clue as to the right answer. It’s a lot of guesswork and assumptions that are all but completely unreliable. He gives the example motorcycle thefts in Germany, which dropped unexpectedly from 150,000 to 45,000. All kinds of theories were put forward in economics, sociology, crime rates, employment trends and so on. The truth turned out to be, of all things, helmet laws. Germany mandated helmets for motorcyclists, which kneecapped the casual theft of motorcycles. Helmets were on no one’s top ten list of causes.

There is a lot on mice. Mice used in experiments have proven to be frustrating for those keen on definitive findings. The same batch of mice, bred to have identical properties, put in identical labs, with identical conditions and food, but in geographically different facilities, have produced different results in identical tests run on them. What is it we don’t know? We don’t know. But no one can rely on test results; that much is certain.

My own favorite story of lab mice concerns the researchers. A study wanted to determine if mice could hide their pain. Researchers injected chemicals into mice legs which gave them a great deal of pain. They found that when female humans handled them, the mice grimaced freely. But when males handled them, they braced themselves and hid their suffering, putting on a brave face. They wouldn’t allow themselves to show weakness before men. This even worked when a male’s used t-shirt was left by the cage. The experiment showed two things, neither of which has to do with muse pain. The results of mice studies are colored by the sex of the researchers and are therefore unreliable. And every mouse test going back a century is invalid because it did not take into account the sex of the researchers who examined them daily if not hourly. Our unintended arrogance in announcing results of such experiments is typical of the hidden half Blastland talks about, even if he doesn’t explicitly cite this one phenomenon.

At bottom, Blastland is saying we are nowhere near as far along as we claim and like to believe. We need a little more humility and a lot less hubris. He quotes Gustave Flaubert, hundreds of years ago: “The rage for wanting to conclude is one of the most deadly and most fruitless manias to befall humanity.” We need to accept the imperfection, not of the universe, but of our knowledge of it. That would change the whole frustration index of continual failures in science.

Meanwhile, back at the marmorkrebs, the crawfish Blastland says have stymied all the experts, it instantly occurred to me while reading the prologue that there is an obvious answer. At least to me. The fact that no males participate in the reproductive cycle means the DNA of the newborns is deficient and therefore unstable. This will produce unpredictably different, if not deformed offspring, with uncertain futures. Parthenogenesis among crawfish is unnatural, and Darwin would posit that situation could not last. Defective chromosomes missing the male input will see to it the subspecies of genetically identical females does not continue. But what do I know. I’m no biochemistry researcher. I just review books.

David Wineberg

[Dr. Dima · Rating 2 out of 5]

The Hidden Half is mainly an argument of two halves: what we know through research, anecdotes, probabilistic knowledge, etc; and what we don't know as a result of hidden messy factors that lie beneath our knowledge. The book uses stories from a multitude of fields such as genetics, medicine, economics, education, and social sciences to make an argument for the limitations and shortcomings of our knowledge caused by the “enigmatic variables” inherent in our lives.⁣
⁣
The book has a prologue, nine chapters, and a postscript. In the prologue, the author sets the scene using an intriguing research study of marmorkrebs (which is admittedly my favourite part of the book). The next 8 chapters delve into topics such as the replication crisis of scientific research, the limitations of big scale probabilities, the complexity of causal interactions, unintended consequences, and other issues. Chapter 9 offers 12 suggestions to acknowledge and address the hidden half, such as adopting multiple paths to knowledge, experimenting, embracing and communicating uncertainty, etc.⁣
⁣
The book has an insightful premise and is certainly thought-provoking. However, I was generally disappointed for several reasons. In one example, the author suggests that a potential cough might’ve saved a long time smoker from lung cancer, which is a suggestion that lacks scientific basis. Furthermore, the author argues (and I agree) that behind the big probabilistic knowledge at the population level, there is a hidden half of enigmatic variation at the individual level, but unlike the author, I see great hopes for personalised medicine and I do not hold the author’s views about this field of medicine. In addition, the first 8 chapters were highly repetitive with a multitude of examples that revolve around the same arguments. In contrast, the final chapter, where solutions are proposed, was relatively shallow and lacking in depth. Overall, the book was not a page-turner for me.⁣

[Ruby Grad · Rating 5 out of 5]

This is a great book, and everyone, especially policy makers, needs to read it. The author tells us, in an easy to read and understand and often humorous way, that so much of what we believe and are told to believe doesn't really encompass the whole picture. We go on to make policy that affects thousands, if not millions, of lives based on what we believe is a solid conclusion, but hasn't taken into account the true nature of life, that everything is ultimately uncertain and we can only do the best we can. If we all took this to heart, there would be far less division and more openness and willingness to allow failure. There would also be far less emphasis on the short term at the cost of the long term. I highly recommend reading this book!

[Jeremy · Rating 3 out of 5]

Probably 2.5 stars, but I'll round up because I agree with the content. The best part of this book was an opening page quote by Daniel Boorstin from Cleopatra's Nose: "The great menace to progress is not ignorance but the illusion of knowledge." I agree wholeheartedly with that quote. The book did not really have much to add from that quote however. The ideas Blastland conveyed throughout the book were not new to me and were not particularly well written. He added nothing to the conversation, but for the uninitiated, this is a fine introduction. Bottom line, we generally know less about the world than we think, probably much less. Don't be overconfident.

[Popup-ch · Rating 4 out of 5]

The inordinate impact of the infinitesimal - It's impossible to influence it all.


Even when we're in total control of everything there are hidden parameters that can have outsized proportions.

Blastland observes that even when we try to control for all discrepancies, there are so many hidden factors that it becomes effectively impossible to reproduce the same conditions twice. This has obvious applications in the Replication Crisis in academia as well as in politics, where it is impossible to show what policy is responsible for what result.

Sometimes there really is a hidden factor - he takes as an example that motorcycle thefts decreased by five sixths in Germany over a single year in 1986. The cause was not better locks, increased penalties or better education, but a new helmet law. Riders without a helmet were routinely stopped, which meant that a spur-of-the-moment theft suddenly came with a serious risk of being stopped by the police. (Very few thieves walked around with a spare crash helmet, apparently.)
But more often the hidden cause remains obscure. As an example, the teenage pregnancy rate has dropped significantly in the UK over the past decade, and none of the proposed causes measures up (increased sex-ed, social media, stigmatization in media, welfare reforms, drug policy, etc. ) It was probably a mix of all these factors as well as several others.

It sounds a lot like chaos theory, where imperceptible changes in the initial state can have inordinate impact on the eventual outcome.

The take-home message is that the world is messy, and that anyone who 'knows' what's about to happen is deluding themselves. There are no silver bullets.

Blastland is a former presenter of More-or-Less on the BBC, so it's only natural to compare with Tim Hartford, who currently wears that hat. Hartford is by far the better writer, though. Blastlands prose is correct and readable, but occasionally verges on the complex and pompous

[Pam Christison · Rating 3 out of 5]

Interesting book on the use and misuse of numbers in what we think we know in economics, science, public policy, and more. Since the foundation of my political philosophy is "I think it's going to be more complicated than that" I am a receptive audience. That said this wasn't really a gripping page-turner of a read for me.

[Dermot Casey · Rating 5 out of 5]

One of the best books I've read in terms of its opening up huge questions about the world and how it works in the best possible way. Questioning of the limits of what we know without falling into pseudoscience its a super read. The first chapter and the chapter on GDP worth it if there was nothing else in the book. And theres loads more in the book.

[RAD · Rating 1 out of 5]

Half Hidden

I wanted to like Michael Blastland’s The Hidden Half, which I purchased on amazon in March 2021. My vocational and avocational interests lie in complexity, data analysis, and modeling, and while this clearly isn’t an academic book, I purchased it on impulse, overcome by confirmation bias from the title, subtitle, and cover illustration—all of which are brilliant. As a natural skeptic and critical thinker, I am completely on board with the concept of imperfect information, or epistemic uncertainty. But the fact that Tim Harford’s name is misspelled on the cover (“Hartford”) should have given me pause.

The book begins with an interesting, though belabored, 13 pages on marmorkrebs (genotypically identical crayfish, but phenotypically different) but loses steam thereafter. Blastland’s book is beset with three primary faults: it is subjective, superficial, and separated.

According to Atlantic Books, the publisher of Half, Michael Blastland is a writer and broadcaster, having created BBC Radio 4’s More or Less program. There is of course nothing wrong with this; there is no claim to the contrary. Under the subheading “Mansions of Straw,” Blastland himself writes, “I’m a journalist, a hack. What do I know? That’s not modesty; it’s the feeling of being intimidated. This is a hard and ubiquitous problem, and we will have little choice but to skate over oceans of scholarship” (p. 17). Caveat lector. At least the reader who doesn’t know of Blastland is given this introduction on the 17th page. In one of the more cogent explications in the book, Blastland goes on to make two additional points: first, he is a journalist, not bound by specialization, and “foolish enough to trample across boundaries” (p. 17). Second, he is “interested in how ideas show up in public debate” (Ibid.). If we are to take that statement at face value, then this book would be a work of intellectual history, as the concept of uncertainty goes back to the ancient Greeks. This is not that book.

Half, then, is written by a journalist unencumbered by expertise (which is fine—even sometimes good). But I lost track of the number of times Blastland uses the first person. Perhaps the most audacious example is on page 182: “…everyone…needs to admit the limits of their probabilistic knowledge…I am past being polite about this.” Why should the reader care about his opinion? Or on the prospect of specialized medicine: “I fear that this talk feeds unrealistic expectations.” (p. 185). Or on the topic of failed businesses: “For my own part, I would like a more diffident public conversation, more frank about what seem to me huge and obvious uncertainties” (p. 214). Doubly subjective here: not just first-person, but not even definitive—these shortcomings only “seem” to be so. Or: “My naïve question is whether uncertain times ought rather to lead simply to…well, uncertainty” (p. 238; ellipsis original). Still more: “My view, consistent with this book, is that it’s highly likely we are influenced by small details, or combinations of them” (p. 130). Now the reader has to contend not only with Blastland’s own subjectivity, but apparently the subjectivity of the book’s author as well.

Superficiality is a pardonable offense in a nonacademic book. But it remains difficult to see such a potentially deep topic diluted with platitudes. In Chapter 1, titled “Bill is not Ben,” and using an example of schizophrenia being present in only one of a pair of identical twins, Blastland muses “in the kitchen of life…you can’t bake the same cake twice” (p. 38). Or summarizing difficulties improving malnourishment in Bangladesh: “as a fine book once put it, everything is obvious once you know the answer” (p. 84)—and this with an endnote! On statistical significance tests: “Of course, all methods can be misused; whether the misuse discredits the test, I’ll leave to others” (p. 140). Why are we still reading?

Superficial sophistry: “In terms of practical policy, Nancy Cartwright told me: ‘Hedge your bets. I’d expect a lot of our social policies, no matter how well thought out, no matter how well intentioned, to fail. That doesn’t mean we shouldn’t try to solve problems, and we don’t do better when we try to solve them than when we just let them drift. But you should be prepared for failure’” (p. 198). Ostensibly this common sense masquerading as deep thoughts are print worthy because they were told directly to the author. Perhaps sensing this, Blastland immediately follows this quote with what appears to be a quote from a book, as both margins are indented:

To establish a general claim, it takes a great deal of to-ing and fro-ing: an interwoven complex of conceptual development, theories—big and small, observation, experiment, analysis, modelling, reasoning, antagonistic assessment and severe testing. A credible general claim will rest on a tangle of support.

And no footnote or endnote reference! We’ve already lost the author. From whence does this quote arise? Oh editor, wherefore art thou?

It gets better: “Governments announce this or that, explain how it will transform life and prosperity with a near-wilful disregard for the unknowns, they pull a policy lever, but it just doesn’t happen like they said” (p. 207). (The reader may choose to disregard the fact that this is a particularly egregious example of a run-on sentence). After cataloging things that “don’t go to plan,” Blastland brings in his statistical guns. “Faced with deep uncertainty and such a patchy track record, you would expect ambition for policy to be tempered with caution or humility. At least that would be evidence we took our limitations seriously. But when I looked at random at just one week of news in the UK, there was little evidence of any such thing” (p. 208). A one-week sample size drawn from the news: we’ve moved from the Hidden Half to the Hidden Asymptotic Limit Approaching One. And—wait for it—we get another endnote for this.

Finally, separated: The Hidden Half is an anglophile’s delight, hiding a large fraction of its 300-page verbiage from the more than half that is not comfortable with the Queen’s English. This is not an American-against-English scrum over “color” or “colour”, “willful” or “wilful”, but rather a whirling dervish of baffling phrases (two instances of “chalk and cheese”); football (i.e., soccer) references (being “sent off” with “red cards”; p. 127); nearly three dozen “UK” references; and exemplars including the sprawling MSAs of Nottingham and Derbyshire. There are epigrams for each of the nine chapters plus the prologue: seven (Dickens, Woolf, Whitehead, Doyle, Mill, Huxley) are English; one (Henry James) is half-English; two (William James, Thomas Pynchon) are American, and one (“possibly” Mark Twain) of somewhat dubious origin, but it could be English! A non-English reader is, shall we say, gobsmacked.

Two concluding anecdotes: first, late in the book, Blastland is comparing an actual mouse trap to the old board game “Mouse Trap” (we’re told that it is for the purpose of explaining causal models). To describe how the board game works, Blastland actually quotes Wikipedia: “To quote Wikipedia, [Mouse Trap] works like this: ‘the player turns the crank, which rotates a vertical gear, connected to a horizontal gear. As that gear turns, it pushes an elastic-loaded lever…'” (p. 246). After photos of mouse trap (the killing device) and Mouse Trap (the board game), Blastland leans on Wikipedia again for another Rube-Goldbergesque quote (pp. 246-247).

Second, Half concludes with a parade of “it” pronouns in search of vague antecedents: “Be as determined as you like. Sometimes it will pay off. But beyond that, chill. It’s not in your hands, or in your head. It’s out there, in a half beyond your dominion. It will do what it will do, and you might never know why” (p. 249).

One star. Half, if I could.

[Honey · Rating 1 out of 5]

A near DNF for me, but persevered through the whole verbose slog of a book. I thought it could’ve been summed in less chapters with more fact than opinion.

I’ve read other books about causality, epistemology, and honing on the signal and the noise, and they had been easier to digest with well presented, evidence based data.

[Alex Megahy · Rating 5 out of 5]

A very good book giving lots of examples (via stories) of the limits of human knowledge. What I gained from this is to always be inquisitive and find out reasons why something might not work.

[Celeste · Rating 4 out of 5]

Yet another book adding to the corpus of general principles some of us are familiar with: humans have a tendency to come up with causal explanation, stories help us make sense of our lives, “expert” predictions are usually wrong, behavioural science, the replication crisis in science, the inherent chaos of life, black swans, etc.

Regardless, the opening of the book really spoke to me:

“Instead of saying ‘I really need to know this,’ we need to say, ‘I can’t know this and how do we manage in a world when we don’t.”

And the ending:

“The value of uncertainty to business is that without it no one would make real money. If every decision were calculable on the basis of a range of ascertainable probabilities, just about everyone would make the same business calculation.”

Here are the main lessons learnt from the book:
1/ The human tendency to speculate on causes and why people turned out the way they are (my fave topic). The case study on a sudden drop in motorcycle theft was very interesting — the real explanation was counter intuitive (ie a law requiring riders to wear helmets) instead of the usual suspects we tend to prefer in our theorising. Yet there are other problems that are not easily as explained, like why sanitation solutions work in one country but fail across the border, or the confluence of factors that explained the sharp drop in teenage pregnancy in the UK. Any one factor might be absent, and the whole theory collapses like a house of cards.
2/ Medical treatments are usually not as effective as they are advertised, learn to read what probability means. So many claims and findings have been based on claims and findings that were subsequently debunked... for those that claim to be the signal, they might be the noise
3/ Big picture statistics drown out the individual realities and effect on people — when people make abstract statements like “free trade and borders are overall good for the economy” — there are losers in the same economic picture, but while they might seem like statistics to us, each one represents a very real and human life. Beckoning the favourite conclusion: this tendency for the IYIs/ elites to only focus on this big picture explains the once-puzzling rise of populism, Brexit, Trump, and so on.

[Dancall · Rating 4 out of 5]

This book could have been called ‘Fooled by Randomness’ or even ‘Incidents and accidents’. It looks at how we think we see patterns in things, but may be completely wrong because so much of the world is still unknown, even in areas like science. ‘We dream of laws and general truths; the practicality is often a patchwork of unexpected anomalies.’
Lots of great examples including the cause of the drop in teen pregnancy in the UK (some argue the case for the impact of Little Britain’s Vicky Pollard), how The Beatles’ success was as a result of multiple lucky coincidences, how the nudges of Behavioual Economics sometimes work, and sometimes don’t, and how no one really understands why Honda motorbikes took off in the US.

[YHC · Rating 4 out of 5]

目录 · · · · · ·
序言
大理石纹螯虾和暗知识 / 001
常识思维vs反常识思维，看似相同，实则不同 / 005
看不到的诱因 / 007
被忽略的细节 / 010
神秘的变异 / 011
如何直面我们的局限 / 015
我们都住在稻草搭建的大厦里 / 018
一、真的有“蝴蝶效应”吗？
人生道路上的潜在影响 / 023
我们没有亲眼见证过，新墨西哥的一只蝴蝶扇一扇翅膀，就能在中国引
发一场飓风的事件。但人生的“蝴蝶效应”却每天都在发生。人生中细微的
琐事也可能在未来引发一场人生风暴，这似乎恰好回应了混沌理论的观点。
用固有知识是解释不通的 / 025
人类相似性的极限 / 035
无限相似，但依然不同 / 047
一只鼻子引发的差异 / 051
经历：第三种影响力？ / 056
二、不断变化的自我
产生信仰和选择的未知因素 / 059
我们也不可避免地要与外界互动——而互动就会带来一个神秘的、多变
的世界。我们的重点不是要辩称人们的表现是绝对无规律可循的。我们想要
强调的，是一些错误观念，就是想当然地把一些看上去明显可以推导出的结
论当作事实——“如果你的医术很好，那你就会一直很好”，而且这些错误
观念很容易产生。
混乱的思绪 / 062
性格是稳定的，但并非永远如此 / 070
三、你以为你以为的就是你以为的吗？
来历不明的差异，颠覆了我们的认知 / 077
我们无从知晓，过去的经验和已知概率对我们而言是警告还是向导，直
到具体事件在它们的时间、以它们的方式揭示出事物的规律，而这些规律巧
妙地打破了那些最聪明、思维最缜密的人们所想象出的规律，或许直到那
时，我们才能有所启迪。
一个薄弱环节 / 081
无法摆脱的困局 / 084
谁在主宰未知世界 / 086
今时不同往日 / 090
即使当时是正确的，也并非真的正确 / 100
正确质疑自��的信念 / 101
时间的神秘报复 / 105
四、掌握了方法根本解决不了问题
研究“发现”发现不了什么 / 111
对知识的渴望可能会带来错误的知识，并可能对最终结果造成伤害，我
们对此应多一份敬畏之心。我们要知道，有的时候，即使掌握了方法，也不
一定能解决问题。
养牛赚钱吗？ / 116
雅努斯的241副面孔 / 122
疑窦丛生的“知识” / 127
一条路，许多路，还是无路可走？ / 133
五、原则其实不实用
大思想与小细节 / 135
千头万绪的生活，绝不会让我们轻易看清它的面貌，我们应当拭目以待。
受害者、恶棍VS看似无关的细节 / 138
非同相性 / 141
机械思维 / 144
定理无法解释一切 / 147
六、宏观微观大不同
概率的潜在局限性 / 153
即使我们准确无误地掌握了某种知识——即使这一知识经过了严苛验
证，可以在不同群体间传播，还有坚实的理论基础，但换个层面，它依然可
能只是一个无根无据的猜想
两种层面的生活 / 155
可以治疗所有人的药物却未必对每个人都有效 / 159
时而明了，时而无知 / 164
“可能有用”比没用更糟糕 / 165
概率的作用 / 167
个体化医学的时代？ / 169
七、一切并非显而易见
隐藏在简单事件中的复杂性 / 173
如果我们把所有的复杂因素都罗列出来，假设我们对每一个因素都了如
指掌，那还怎么解决争议呢？当每个答案和它的对立面都看似很有道理时，
这种显而易见的论证就是错误的。
“显然，因此……”的误区 / 175
不必去寻找答案 / 177
不存在可借鉴的成功案例 / 183
蓝狐之谜 / 187
八、“例外”才是决定性因素
不可忽视的暗知识 / 191
常识既能帮助我们理解世界，也会削弱我们对世界的理解能力。人类认
知对于混乱无序生活来说只是“冰山一角”，那些不曾了解的“暗知识”也
许会帮我们看清事物的本质。
哪里可能会出错？ / 196
危险的智慧 / 201
例外的复仇 / 203
九、如何突破思维定式 / 211
不同的情境需要不同的策略。没有一定能奏效的方法。但建议往往是值
得一试的。而“尝试”或实验本身就是最好的方法
后记
为何要探讨暗知识？ / 229


后记 为何要探讨暗知识？
暗知识——这⼀概念源于⼀种感觉，即认为这种未知的因素似乎
并不能完全⽤偶然性来解释。我的意思是，许多著作的主题都是探讨
偶然性和随机性——⼤多数都是统计学家们警⽰说我们低估了偶然性
的影响——但即便如此，偶然性依旧是研究领域被忽视的课题之⼀。
学者们⽆数次地试图提醒⼈们，⽽⼈们⼜⽆数次地⽆视这些提醒，这
两个事实使我们不禁要问：为何这些提醒没有得到重视，是否有其他
⽅法让⼈们醒悟？
我怀疑我们⽆法成功（因为我也尝试过）的⼀个原因在于，偶然
性是⼀个抽象概念，⽽“原因”则是⼀个具体事件，它讲述的是某件事
导致了另⼀件事的发⽣，这源于我们最深层的本能。当专家们警⽰
说，某件事可能由于偶然性⽽导致另⼀件事的发⽣，这实际上是在试
图⽤抽象概念说服⼈们放弃对现实事件的信念。这就像试图⽤⼀个想
法去打开⼀罐果酱。例如，有⼀个⽼笑话说：“永远不要试图跟⼀个商
界成功⼈⼠谈运⽓。”真实事例所展现给我们的，是有⾎有⾁的⼈物、
时间箭头、有始有终的情节、各种名字、戏剧性的转折、情感、希
望、失望、成功、强⼤的主观能动性，最重要的是⼀条清晰的因果关
系线。在许多⼈的想象中，相较于抽象概念，真实事例显然拥有更⼤
的说服⼒，这是⽏庸置疑的。
那么，我们应该如何戳穿这些所谓的因果故事呢？隐藏的那另⼀
半试图以它⾃⼰的⽅式使它们不攻⾃破。它营造了这样⼀个世界，那
⾥充斥着⽆数相互⽭盾的细微因素，然后，它把这些因素想象成⼀个
个独⽴事件。这样⼀来，这些神秘的细微因素就成了转折点，在它们
的影响下，那些处于微妙平衡的各种因素，就会使事件朝着不同⽅向
发展。⽆论是⻘少年犯罪率、GDP、⼤理⽯纹螯虾的最终形态，还是
孟加拉国的⼀个援助项⽬，通过展⽰这些故事的结局是如何因为某个
神秘变量⽽发⽣反转，偶然性在这⾥就被具象化了。本书将这些神秘
的变量具化为⽣动的、客观的物质存在，如婆婆、复印机、公交⻋、
⼀声咳嗽、⼀句“敢辞职，就不要进家⻔”的指责，或者其他精确⽽特
殊的经历。在这个世界⾥，这些被物化的因素会受到各种纷繁细节的
影响，发⽣微妙变化，继⽽导致每个故事都极易产⽣另⼀个完全不同
的结局。在这些所谓的因果事例中，我们倾向于把⾃⼰看作⽆所不知
的⼤英雄，假如我们想推翻这些事例，使⼈们相信各种知识的易错
性，或许最好的办法就是去找出更多的事例。
这些颠覆性的例外事件⽆法成为可效仿的模板或范例，但却反映
出了所有的事例、理论和所谓原理的特质。你不可能靠⼀声咳嗽来预
防癌症，就像你不可能靠⼀台复印机来脱贫⼀样。这些事例不是证
据，它们的出现只是在指责我们将知识扩展到了过于宽泛的领域；它
们既不是规则，也不是原理，但它们是⼀种危险信号，提醒我们规则
存在局限性。
简⽽⾔之，暗知识所讲述的故事，建设性地激励我们更谦卑⼀
些，不要⽤寻找规律或试图控制的想法来填补⾃⼰的想象⼒，⽽是应
该去发掘每⼀个颠覆性的细微经历，从⽽时刻提醒我们那些隐藏在细
微因素中的偶然性。
⽆论如何，这是我⼀直在努⼒做的事情。这么做有⻛险，尤其是
可能会被有意或⽆意地曲解，认为既然⼀切如此变化⽆常，那我们根
本⽆法做任何事。我完全反对这种观点。
对潜在现实进⾏先知性的描述，不是⼀种成就，⽽是⼀种幻想。
它是⼀颗落⼊池塘的⼩⽯⼦。也许它⽆法激起任何波澜。但假如正相
反呢，谁⼜能知道呢？


让我们陷⼊困境的不是⽆知，⽽是看似正确的谬误论断。
⻢克·吐温
20世纪90年代，在德国⽔族圈⾥流传着这样⼀条谣⾔：⼀种从未
被科学界所发现的奇特新物种出现了。由于⼈们从未在野外发现过这
种⽣物的踪迹，所以也就没⼈能确定它是如何出现在德国⽔族馆中
的。前⼀天它还不存在，可第⼆天它便出现在了⼀只⻥缸⾥。
这种后来被命名为⼤理⽯纹螯虾的⽣物，是⼩⻰虾的⼀个新品
种。它们与其他⼩⻰虾⼗分相似，只不过有⼀个显著的区别是：雌虾
⽆须受精，就可以⾃发产卵并孵化出幼虾，这个过程被称为孤雌⽣
殖。也就是说这些⼩⻰虾不需要交配即可繁殖，其后代都是天然的克
隆体。
表观遗传学家弗兰克·利科说：“⼈们诧异不已，只有雌虾，雄虾在
哪⼉？”他还补充说，⼀个新物种的进化通常需要数千年的时间。
其他任何⼩⻰虾或相关⽣物，包括螃蟹、⼩虾、对虾在内的近
15000种⼗⾜⽬甲壳纲⽣物都没能做到这⼀点。似乎没有⼈知道⼤理⽯
纹螯虾是怎样出现的，只能猜测某⼀天，在某个⻥缸⾥的⼀只⼩⻰虾
⾝上，突然发⽣了⾃发性的基因突变，⼤理⽯纹螯虾——“夏娃”便诞
⽣了。
所有这⼀切已经⾜够奇特精彩了，但故事还在继续：在它们的突
然出现震惊了我们之后，⼤理⽯纹螯虾还引发了⼈们的诸多猜测。
这种螯虾⾸先引起了科学家们的注意，于是他们在2003年的《⾃
然》（Nature）杂志上发表了⼀篇简短的报告，以叙述故事的⽅式向研
究界正式介绍了⼤理⽯纹螯虾。
有传⾔称，出现了⼀种不明⾝份的⼗⾜⽬甲壳纲⽣物。该⽣物是
⼀种带有⼤理⽯花纹的⼩⻰虾，地理来源区域不明，于20世纪90年代
中期被引⼊德国⽔族圈交易中，据传该⽣物能够单性繁殖（即孤雌繁
殖）。我们在此证实，这种⼤理⽯纹螯虾在实验室条件下确是孤雌繁
殖。
克隆功能使⼤理⽯纹螯虾成了⾃然界的威胁，它们拥有极强的侵
略性，释放⼀只就能建⽴⼀个完整的种群。研究表明，不仅如此，它
们还“强健⽽多产”：成熟快，产卵多。2018年，弗兰克·利科宣称：“只
要将⼤理⽯纹螯虾放⼊你的⽔族箱中，⼀年后你就能拥有⼏百只了。”
此前，⼤理⽯纹螯虾的后代四处横⾏，在⻢达加斯加泛滥成灾，⼀度
引起热议，更是成为头条记者争相报道的新闻：“变异⼩⻰虾的⼊
侵”。
然⽽，这些克隆⽣物却对科学研究具有更重⼤的意义：⼤理⽯纹
螯虾可能有助于解决“基因与环境之间到底谁在起作⽤”这⼀古⽼⼜棘
⼿的难题。因为研究⼈员意识到，他们⽆意间发现了⼀个理想的实验
对象。
通常情况下，想要梳理出事物的成因并⾮易事。假如你患上了⼼
脏病，那么潜在的病因会有很多，既有基因⽅⾯的，也有环境⽅⾯
的，可能与遗传、饮⻝、锻炼（过少或过多）、压⼒等多重诱因有
关。⽽克隆体可以确保基因的作⽤不变，从⽽更容易梳理出其他影响
因素。当把⼤理⽯纹螯虾两两相⽐较时，⽆论它们发⽣了怎样的变
化，都不会是由于纯粹的基因差异导致的。这些克隆⼩⻰虾是天赐的
完美实验对象。
于是有⼀天，德国的研究⼈员选择了两只雌虾作为亲虾，它们分
别成了两个实验谱系的伟⼤⺟体，并被命名为A和B（当你是⼀只⼩⻰
虾时，即使是只传奇⻰虾，也只能得到这样的名字）。研究⼈员将A和
B的后代放⼊⽔箱进⾏观察。⾃然，这两个谱系的⼤理⽯纹螯虾具有完
全⼀致的基因。这不是假设，它们经过检测，是符合基因⼀致性的。
但研究远不⽌于此。这些⼤理⽯纹螯虾还在完全相同的实验室环
境下孵化⽣⻓。如此⼀来，它们⽣⻓过程中的每⼀种影响因素都尽可
能地保持⼀致。它们被投喂相同的⻝物（既然有⼈问，那就顺便提⼀
下，投喂的是德彩薄⽚混合型饲料[1] ），定期接受疾病检查，被饲养
于装有⾃来⽔的简易⽔箱中，且⽔温与室温保持⼀致，甚⾄每次都安
排同⼀位研究⼈员对它们进⾏检查。这样做的⽬的，是尽⼒消除⼀切
我们所能想到的变量。可以说这些螯虾从⼀出⽣，就住进了⼈类设计
出的、完全相同的环境中。
那么，这些克隆⼩⻰虾会⻓成什么样呢？花点时间，⼤胆猜测⼀
下：应该都⻓得⼏乎⼀样，或者完全⼀样吧？
毕竟，我们明⽩，我们所了解的关于这些⼩⻰虾的⼀切都是可知
的，⽽且就我们所掌握的因素⽽⾔，它们每⼀只都是相同的。基因和
环境是主宰⽣命的两股强⼤⼒量，也是⼈类认知领域内的两座⼤⼭。
⼆者之间存在着⼀场有关解释效⼒的永恒战争，⽽作为实验对象的所
有⼤理⽯纹螯虾在这两⽅⾯都是完全⼀样的。
但是你瞧，图1-1显⽰的是在实验室饲养的⼀组⼤理⽯纹螯虾，它
们全部取⾃同⼀窝卵。这张图出⾃德国实验室⼯作⼈员所撰写的⼀份
研究报告，并于2008年对外公布。它成了遗传学领域引⼈瞩⽬的发现
之⼀，这份殊荣也的确实⾄名归。显然，这些⼤理⽯纹螯虾各不相
同。在同等条件下饲养的这些“⼀卵同胞”中，有些螯虾的尺⼨竟然是
其他同胞的20倍。
这些螯虾之间可⻅的差异是惊⼈的，尺⼨⼤⼩只是其中最为明显
的差异。在这数百只作为研究对象的⼤理⽯纹螯虾中，每⼀只⼤理⽯
纹螯虾都是独⼀⽆⼆的。它们的感觉器官和内脏器官都存在明显的⽣
理差异，活动和休息的⽅式也不同：有的躲在遮蔽物下不动，有的则
仰卧着。这些螯虾的另⼀个巨⼤差异是寿命，从437天⾄910天不等。
它们开始繁殖的时间也有早有晚，产卵数量和次数也千差万别。有的
螯虾⼀边产卵⼀边进⻝，其他螯虾则不然。它们有的在早晨脱壳，有
的则在夜晚。

[Paul Sand · Rating 5 out of 5]

[Imported automatically from my blog. Some formatting there may not have translated here.]

I put this book on the get-at-library list after listening to the episode of the Econtalk podcast where the author, Michael Blastland, was interviewed by host Russ Roberts. I'm somewhat embarrassed to admit that I didn't recognize Blastland as the co-author of the excellent book The Norm Chronicles, which I read back in 2014. This book is really good too. Blastland is a journalist (but a smart one), and his prose is lively and accessible.

Here, he takes a hard look at the concept of "all other things being equal". A concept so ancient, it's sometimes expressed in Latin: ceteris paribus. (This page says that Cicero used it.) Although most common explicit use of the phrase seems to be in economics, the concept underlies a lot of science. And for that matter, a lot of life.

The problem being: how can you assume "other things being equal" when they so often are not?

Blastland opens with an unexpected example: the marmorkreb, a species of crayfish. They are parthenogenetic, with all offspring being genetically equal females to their mother. A few years ago, German researchers decided to raise a batch of marmorkrebs with identical environments as well. And yet, their marmorkrebs defied their genes and upbringing, and became unexpectedly diverse. Their size varied greatly, as did their coloration. They socialized with other marmorkrebs differently; they had different lifetimes; they had different eating behaviors; …

It's almost as if they were individuals, not simply mass-produced clone crayfish. And if you can't assume ceteris paribus with a bunch of clones, how can you assume it elsewhere?

Blastland answers: you often can't, and you shouldn't. Simple mental models of how things "should" work are often correct. But just as often (about half the time?) they fail, because of underlying complexities and confounding details that you didn't consider.

From there, Blastland takes a wide-ranging tour of how that works (or doesn't). Many stories, most interesting. There's (for example) sinful boxer Mike Tyson, compared and contrasted with his sainted surgeon brother Rodney. Tons of research studies that turned out to be irreproducible.

There's way too much to try to summarize, but I found one issue Blastland raises particularly interesting, and it brings in his Norm Chronicles co-author, David Spiegelhalter: studies of "risky behavior" based on large sample populations can be (and often are) reported misleadingly. The semi-amusing example was from the Lancet where the well-documented result was that there is "no safe level" of alcohol consumption. Even one drink per day raised your risk of developing a serious alcohol-related health problem. And the article suggested that public health institutions should “consider recommendations for abstention”.

Let's swing over to Spiegelhalter's Medium article that Blastlad cites:

Let’s consider one drink a day (10g, 1.25 UK units) compared to none, for which the authors estimated an extra 4 (918–914) in 100,000 people would experience a (serious) alcohol-related condition.

That means, to experience one extra problem, 25,000 people need to drink 10g alcohol a day for a year, that’s 3,650g a year each.

To put this in perspective, a standard 70cl bottle of gin contains 224 g of alcohol, so 3,650g a year is equivalent to around 16 bottles of gin per person. That’s a total of 400,000 bottles of gin among 25,000 people, being associated with one extra health problem. Which indicates a rather low level of harm in these occasional drinkers.

In short: yes, drinking alcohol is risky. But on the individual level the additional risk is small. To repeat: in that population of 100,000, all imbibing one drink per day, four of them would develop a health problem due to their booze consumption.

Spiegelhalter comments:

But claiming there is no ‘safe’ level does not seem an argument for abstention. There is no safe level of driving, but government do not recommend that people avoid driving.

Come to think of it, there is no safe level of living, but nobody would recommend abstention.

It's amusing, sure. But note that this type of research is used to advocate for "public health" measures including taxes, regulations, and prohibitions. It's all fun and games until somebody gets coerced.

[Ronald J. · Rating 4 out of 5]

I thoroughly enjoyed this book. It takes the entire nature vs. nurture debate to a new level. Perhaps both nature and nurture only explain one-half of the differences between us humans. There is a hidden half that we don’t yet understand. Michael Blastland provides many examples of the hidden half existing in many facets of life. He starts with the most fascinating one. In the mid 1990s aquarium owners and traders in Germany discovered a new beast, previously unknown to science. The marmorkrebs, as they were later named, a new species of crayfish. Except there were no males. The females produced clones of themselves, a researchers dream. This is how Blastland explains it:

"Among identical batch-mates in the same conditions, one crayfish grew to be twenty times the weight of another…differences in internal organs, differences too in how they moved and rested. Another big variation was lifespan, which ranged from 437 to 910 days…more differences in how they socialized. [Some] fell into hierarchies, some submissive, some dominant. Some were solitary, others liked a crowd. They were physically different and behaviourally different.

"Normally, we say that if it isn’t genes it must be environment; if it isn’t environment it must be genes. But this seems in some way to be neither.

"This is what I like to call the shock of ignorance. It’s a good moment, a forced recalibration. It reminds us how easily we can be satisfied with established ideas, and what amazement might lie around the corner. If even clones in the same environment are not the same, owing to the power of intangible variables, how reliably can we expect to pinpoint the sources of difference between people, businesses or policies, in all their infinitely messy complexity?"

I liked the discussion on how evidence can be distorted based on the measurement. For instance, rather than measuring cost and benefit over a full year, how about over three years? “One prominent critic of research standards, Professor John Ioannidis of Stanford University, calls it ‘the Janus effect’: evidence is often two-faced.” Consider a country where 100% of the population smokes? How would you diagnose causes of lung cancer without a comparison? There are excellent examples from crime and medicine, to the economy and random controlled trials.

The point is not that we don’t, or can’t, know anything so we should give up. The objective is to learn to live with our uncertainty, and be humble about our knowledge. The book made me think of the philosopher who said knowledge is just postponed ignorance. It also made me think about George Gilder’s Information Theory of capitalism, and how creativity and innovation always take us by surprise, otherwise they wouldn’t be needed and socialism [planning] would work. “There’s no evidence from the future, that’s for sure,” as Blastland write. Would we want to live in world that was certain? Would you like to know the date of your death?

Fantastic read, highly recommended. Also, the Blastland did an interview on my favorite podcast, EconTalk, hosted by Russ Roberts: https://www.econtalk.org/michael-blas...

Michael, if you read this, we’d love to have you on our show, The Soul of Enterprise: Business in the Knowledge Economy. We’ve had Russ on as well, along with many other authors.

Notable
The great menace to progress is not ignorance but the illusion of knowledge. --Daniel Boorstin, Cleopatra’s Nose: Essays on the Unexpected

Is there a relationship between consistency and rationality? Consistency can be over-rated. Economist John Kay pointed out, you can be consistent in believing there are fairies at the bottom of the garden. Consistency alone can be a measure of insanity. Ralph Waldo Emerson’s line: ‘Foolish consistency is the hobgoblin of little minds, adored by little statesmen, philosophers and divines.’

Hayek, Nobel Prize lecture in 1977 was called ‘The Pretence of Knowledge’. He said: “In the study of such complex phenomena as the market, which depend on the actions of many individuals, all the circumstances which will determine the outcome of a process. . . will hardly ever be fully known or measurable.”

Samuel Johnson said: “people are less in need of being instructed than reminded.”

Stories are powerful precisely when they show us the limits of a general claim. The purpose of a story is not to pile on evidence in support of one theory or another but rather to shine a spotlight on an anomaly, like the marmorkrebs – that forces us to say ‘except. . .’”

‘If it looks like a duck, quacks like a duck but needs batteries, you probably have the wrong abstraction’: apparently known in computer science as the Liskov Substitution Principle.

All models are wrong but some are useful (originally by a statistician, George Box). I think the ‘some’ in ‘some are useful’ is probably not as large a proportion as we might hope. “The utility of models can be easily exaggerated.”

John Pullinger, the UK’s National Statistician, put it in recent remarks to the Royal Statistical Society, of which he is a former president: “We need to understand the limits of our own cleverness.”

[Mike Cheng · Rating 4 out of 5]

At its core, this book is about our ignorance on the subject of causality. There is an unfathomable amount of information and intricate complexities behind even the most trivial matters, yet we (as individuals and collectives) assume that we know most if not all that there is about a great many things. It isn’t until when outcomes fail to meet expectations that we, for a brief and fleeting moment, admit ignorance. Even then, however, we are somewhat dismissive of these unknown factors, ascribing terms such as “luck”, “noise”, “acts of God” so that we can continue having the perception of control and understanding.
Author Michael Blastland opens the book with an example of the marmokrebs, a species of crawfish that seemingly and spontaneously were able to reproduce without needing to mate (i.e., parthenogenesis). One would expect that identical genes and the same lab controlled environment would give researchers a blank slate to work with. What happened was quite surprising: the marmokrebs were vastly different from each other. Some were 20x the weight of others. They varied in each having unique markings, as well as noticeable differences in their respective organs. Their lifespans also varied, as did whether they were submissive or dominant, and whether they were loners or gravitated toward the herd. Although genes and environment play huge roles in why organisms differ, how does one explain the wide disparities observed in the marmokrebs? This sets the table for Mr. Blastland’s premise as alluded to above, namely that there are hidden factors and influences at play, so many that seeking knowledge of such is a Herculean, if not Sisyphean, task. Mr. Blastland also makes clear, though, that there should not be a sense of nihilistic despair but rather just simple humility about the illusion of knowledge. Moreover, he also recognizes that even some knowledge brings us that much closer in furtherance of truth, and importantly that the book is not an excuse to blithely resist evidence one disagrees with; to the contrary, one should seek more robust evidence. In other words, the goal is not cynicism but a call to do better - recognize limitations, tread carefully, and test vigorously.
The first third of the book discusses how ignorance results in failed applications across space (what works here might not work there) and time (what worked yesterday might not work today). The second third is about the failures of research and of applying general (and often abstract) principles to single specific instances. Relatedly, even if research has reliably provided data about groups, it still does not sufficiently translate to the individual level. Mr. Blastland gives an example of a study in 2017 about ibuprofen and the pernicious effect on heart disease. Allegedly, regular use was reported to increase the likelihood of a heart attack by 30%. 1 in 800 people were said to be at risk, meaning that the 30% increase meant an additional 0.3 persons in 10,000. What this means is, maybe, that while someone is going to be affected negatively by regular ibuprofen use, for most everyone else it may not make a noticeable difference. On a much larger scale, government will often tout the benefit of some policy that could result in some significant percentage increase, but the important thing to understand is how many will actually be affected in exchange for the cost, whether as spending or reduced economic output.
The book concludes in Chapter 9 with Mr. Blastland’s ‘what to do’ in light of our vast ignorance: (1) experiment and adapt - but remain cognizant that the factors we remain ignorant of could be the critical ones; (2) resist the urge to conclude, and also do not equate confidence with competence (interesting side note: global warming proponents have suffered a loss of credibility by refusing to admit uncertainty); (3) probability is a tool, not an answer nor excuse for ignorance; and (4) relax and embrace uncertainty - know that there are some things that are impossible to know.
Finally worth stating is that the ignorance of knowledge is a reality but not an insult. Human beings are designed to understand the world in the form of narratives, and it would be impossible to navigate the world without some overconfidence, such as being sure that the sun will rise tomorrow.

[Justin Drew · Rating 5 out of 5]

Wow, what a book. This book begins with the quote “it ain’t what you know that gets you into trouble, it’s what you know for sure that just ain’t so“ which is possibly by Mark Twain. The book talks about a new species of crayfish known as the marmorkrebs, similar to other crayfish apart from the fact that loan females collate and hatch eggs without fertilisation. Therefore the young marmorkrebs are genetically the same and when raised in exactly the same lab controlled environment where they have access to the same food and all they want and are regularly checked for disease, they should in theory all be exactly the same. However, when the same genes were raised in the same environment something odd happened, every single one had a unique pattern of marbled markings and had physical differences in their sensory organs, internal organs, how they moved and rested, different life spans ranging from 437 to 910 days. The onset of reproduction was different as was the quantity of the eggs and the number of batches. Same genetic coding, same environment and yet all were different. There was something else, a hidden something, that made them different. What was the cause? We might wonder about genes or environment, or nature and nurture, but what are other factors that make us all different and what is the impact upon research papers which tries to control all variables. We need to accept the possibility that the big influences are not as orderly or as consistent as we expect and that the way things turn out is not necessarily bound by observable laws or common factors, but also by massive uncommon factors. This book explores three arguments. The first is that we need to face up more readily to the many mysteries and surprises about how human understanding works. We need to accept that we know less than we think we know. The second idea is to put aside questions of rationality and dig into the hidden half of enigmatic variation instead of the positive force of disruption. We dream of laws and general truth when the practicality is often a patchwork of unexpected anomalies. The third idea is how do we cope in a world where we know less than we think and that the world falls into lines less readily than we suppose. To accept that we live in houses of straw rather than in houses of brick.

There is a story that I love about a father who was a criminal, neglected his two sons, gambled and was an alcoholic and ended up in jail. He had two children, one was respectable, married, and a good career and was of a good nature. The second son was exactly like the father. Both sons, when asked separately by a psychiatrist why they had ended up that way, both said “with a father like mine, what do you expect?“ We can use narratives to explain cause-and-effect, but are these true? Mike Tyson, the boxer, was brought up in serial homes in condemned buildings with a heavy drinking mother who could be violent, took cocaine when he was 11 and had been arrested 38 times when he was 13. He finished up in a correctional institution for boys in New York's most infamous juvenile prison. Is it any wonder that Mike Tyson became a self-styled bad man, bit off another boxers ear and ended up in prison for rape. With an upbringing like that how else was he due to turn out. However, the anomaly of this story is that Mike Tyson had a brother, Rodney Tyson, who was also raised in a similar situation to his brother. Yet he became a specialist surgical assistant in a hospital trauma department in Los Angeles where he also helped patch up the victims of crime. Same background yet both brothers became so different. One was a violent and angry man and one had a vocation to help. Similar backgrounds yet one ended up in prison and the other brother ended up with empathy and compassion. Does the narrative predict and the stories we tell ourselves predict how everything will end up. The author talked about a research paper called ‘shared beginnings, divergent lies: delinquent boys to age 70’. Two researchers followed up a study of 500 men who have been in serious trouble in childhood and another 500 who despite similar backgrounds had not. They tracked down as many of the men in the original study and the one thing they found was that everybody brought up in a similar environment had ended up with very different lives and careers. Some remained antisocial but many led appropriate and good lives.

When we look at studies of twins, both identical and those who are born conjoined, we know that even these babies and children grow up to have very strikingly different aspects of personality. The effects of your environment including your parents do not seem to have as much of an influence as we believe on how we behave. There would appear to be chancing counters which can dramatically impact and change events in your life and how you end up, it might be a cough that causes cancer or a chance encounter that changes your life. If you consider that 50% of our personality is mainly shaped by genetics, there is 50% that we don’t know. But even when we look at ourselves we are not symmetrical. Draw a line between our face, one side will be slightly different to the other. A woman who gets breast cancer in one breast, will not necessarily get cancer in the other breast (it is almost similar to a woman who has never had breast cancer). In women over the age of 50 it’s 13% as opposed to with a lifetime risk of breast cancer overall is 12%. There is an interesting experiment described in the book where people were made to decide choices, for example when asked which of several girls in pictures they fancied, when later pictures were switched and they were told that they had said they fancied a different girl, they were then asked to explain the reason why even when they weren’t aware that they had been given a different choice. They would tell themselves a story or confabulate (to find a plausible answer to why we behaved in a certain way) something to explain their reasoning, even when it wasn’t correct.

We often have conflict free and inconsistent belief systems, even when we have thought about them. The world is often too complex and difficult to understand so once we form a belief about a story that we’ve told ourselves, we can be very resistant to any change in that story. We create frameworks and stories to explain things which aren’t necessarily true or correct. If you ask researchers to find out the answer to a simple question such as how many red cards might be given to 2 pairs of people of different colours to determine if referees are racist, you will come up with a wide variation of approaches that lead to a wide array of different answers. The greatest obstacle to learning more understanding isn’t necessarily the wisdom behind it but what we think is so and all the stories we tell ourselves rather than the truth.

There is an interesting story about a chicken and a farmer. The chicken is fed by the farmer every day and the farmer comes to feed the chicken which comes to eat the food. One day the farmer comes along just before December and rings the chickens neck. Just because you think something is going to happen doesn’t mean to say it’s always going to happen in the same way. As Bertrand Russell once said “the mere fact that something has happened a certain number of times causes animals and men to expect that it will happen again“. We should remain uncertain about everything we believe. We often look for a silver bullet to explain everything but often a range of complexities such as why teenage pregnancies have dropped or how schools can work more effectively in making children’s learning more effective. Sometimes there will be a mirrared of reasons and looking for one simple reason to explain cause-and-effect can become our downfall. If you want to have a laugh, just look at all the predictions that people make that are often wrong. Everyone thought China was going to remain a poor country, no one predicted the fall of the Berlin Wall and the end of communism. People can have genetic mutations which means some people can’t handle alcohol whilst others can, some people can have an anaphylactic response to nuts but others can just devour them all day long.

The book ends with a number of approaches for things that we can do to help manage uncertainty and complexity and summarised in the quote “Life’s single lesson: that there is more accident to it than a man can ever admit to in a lifetime and stay fine” by Thomas Pynchon. One of these is to embrace uncertainty. We don’t want to know what day we will die and uncertainty makes life interesting.

[Rory Fox · Rating 5 out of 5]

Excluding particularly virulent conditions, the chances of getting breast cancer in a second breast are less than 20%. Why is that, when both breasts have a similarity of genetic and environmental background?

With questions like this, the author explores many examples of inexplicability within Science. Marmorkrebs have identitical genetics and (as far as we can tell) environmental factors, yet the author shows a photograph illustrating enormous differences between individual Marmorkabs. How is this possible?

The thesis of the book is that Science is a lot less sure than it sometimes claims to be, so the golden rule should be ‘be humble’ (62%).

To justify this, the author describes the crisis of replication within Academia. He cites a 2012 study which found that Scientific results published in prestigious journals could only be replicated in 6/53 instances. That looks like a 90% failure rate and it means that anyone trusting the research is potentially wrong to do so.

Whether it be contradictory aphorisms about the wisdom/stupidity of crowds, or the role of chance, the author shows that there are a myriad of hidden variables at work. For example, Charles Darwin’s discovery of Evolution may have depended on the completely random fact of the shape of his nose. When Darwin was interviewed by Robert Fitzroy, the captain of the Beagle, he used his belief in nose-shape as a sign of personal character, to help him select who was to accompany him.

Sometimes it is the complexity of interpretation that is the issue. The author cites a study of racism in sport. 29 researchers looked at the same data on players given red cards. 9 teams reported that Black players were discriminated against. The other 20 researchers found different outcomes, with 2 teams even finding that it was White players that were discriminated against. How can researchers reach completely different results, on the same basic data? The answer is that there are so many factors at work in the data, that there isn’t any such thing as a finite closed set of data. This means researchers can follow different threads to totally different conclusions.

The author isn’t sceptical or negative about Science. He just thinks that people need to be careful not to push it further than it can actually go. Some of our problems are due to ignorance, some due to motivated practices and some to the sheer complexity of factors at work in the world. Whatever the factor, we need to treat Science with respect and that means not asking more of it than it can deliver. To use an old idea: if we are to be ‘virtuous’ with data, then we need to remember that virtue all too often lies in the golden mean of avoiding extremes

Overall an enjoyable read with lots of examples to illustrate a thesis of intellectual humility that we could all benefit from remembering.

[Marco · Rating 3 out of 5]

The book contained some important insights that made it worth my time. There are two main issues that I take with the book. 1. Logical fallacies. 2. Other frames could have been just as powerful.

1. There are many logical inconsistencies and straw men that make the overall foundation very shaky, which is ironic for a work that talks about the value of thinking rigorously.
Some highlights:
- The marmorkrebs story (a type of small crayfish) is used as an illustration for how we cannot make predictions about how things will play out, even given the exact same initial conditions
- At various points, the author says that "what is here is not there", a law that applies in one context does not translate to another one; or it does not apply at different scales, at different times, etc. The point is made in a convincing manner
- A few times throughout the book, the author uses the marmorkrebs story to illustrate how other things should be subject to the same causal opacity and indeterminacy. But if "here is not there", why is the author so confident in assuming that all the world is a marmorkreb?

2. Other frames could have supported the stories just as well. While most of the anecdotes and examples are interesting in their own right and one can learn from them, the book could have used other lenses to look at the world that would have equipped the reader to make meaning of it all.
- Complexity. You could make sense of the examples by looking at how the increasing number of variables and the increased opacity in our knowledge of fundamental laws of causality.
- Some disciplines are foggier than others. Physics experiments do not tend to suffer from the same Janus effect because we ask a question and nature responds in a clearer way.
- Context dependence. Our capacity to predict is sharply reduced as we move outward from a laboratory that is isolated from the world, all the way to a vast entangled jungle where everything is connected to everything else.
- Decision making and risks. The book touches lightly on this, but an interesting angle could have been: What do we do then? The world is unpredictable, how do we make decisions in light of this uncertainty?

[John Elliott-White · Rating 5 out of 5]

I enjoy any book that pulls the rug from under conventional thinking and forces us to view the world with fresh eyes. The central theme is the assertion that in every field of human knowledge or endeavor, we may well identify a handful of causal factors which are key in determining outcomes ... but it's only half the story. There is often also a whole host of micro-influences which, in different combinations, can produce very unexpected results. This is the hidden half.
And these micro-influences may not just be unknown, but may even be unknowable.
Unfortunately, whether by vanity or ambition it is a human tendency for those who are considered - or consider themselves to be - authorities, to sweep these factors aside in favour of publishing or proclaiming their theories with a degree of certitude that is entirely unwarranted.
Having set out his stall, Blastland explores multiple examples from many different arenas; although to cover such a broad scope, these are not deep dives. He rounds off with some suggestions of how we might live more comfortably with uncertainty – authorities and laypersons alike.
Critics of this work point out that Blastland is not a scientist (how dare he!), other suggest that what he is saying is trite and obvious and does not need saying. But it does. Repeatedly.
This is a populist book to highlight these issues for us laypersons who are taken in over and again, either by the absurd overconfidence or overselling of ideas by those who really ought to know better. But it should also serve as a reminder to politicians, scientists and other authority figures. If they did a more responsible job of acknowledging and communicating uncertainty - not that I'm holding my breath - a book like this would not be necessary. But it is.
(Note: my review is based on the Audiobook which is not listed here)

[Scott Muc · Rating 4 out of 5]

Another book I read because I heard the author interviewed on Econtalk: https://www.econtalk.org/michael-blas...

The book didn't disappoint and helped me expand upon the brief tidbits dropped during the interview.

The structure of the book felt quite natural and built upon the chapters preceding them. The first chapter about the genetic clones of crayfish felt like a good way to begin. It brings the concept of "The Hidden Half" right into the foreground. If someone has identical genetics, and the environment was kept consistent as humanly possible, what could possibly account for the variations?

The "What to do" chapter was reasonable good at communicating some methods of managing this world of uncertainty. Suggestion #9 (Don't use probability to disguise ignorance) was impactful to me. After reading Thinking in Bets, I had swung quite far in that direction. This suggestion is something I'll need to remind myself of in order to ditch the bet idea and simply state I don't know and need to learn more about the topic. Funny enough, much of what is mentioned in this chapter describes Agile software development (when followed with discipline and purpose, not mechanically).

This was a fun follow up after The Drunkards Walk and Thinking in Bets. Unfortunately it's left me feeling cynical that any public policy conversation will continue with deterministic promises and growing fear of the uncertain.

[Jilly · Rating 2 out of 5]

An interesting read which got me thinking which is always a good thing. Rather worrying though that so much of what we think we know is not actually what we think we know! It seems that there are no certainties in life, only probabilities and chance and it all depends on so many variables that we will never be able to see the full picture. Hence the hidden half.

I have only given it 2 stars just because it was really hard going in places with so much data.

*********************************

Why does one smoker die of lung cancer but another live to 100? The answer is 'The Hidden Half' - those random, unknowable variables that mess up our attempts to comprehend the world.

We humans are very clever creatures - but we're idiots about how clever we really are. In this entertaining and ingenious book, Blastland reveals how in our quest to make the world more understandable, we lose sight of how unexplainable it often is. The result - from GDP figures to medicine - is that experts know a lot less than they think.

Filled with compelling stories from economics, genetics, business, and science, The Hidden Half is a warning that an explanation which works in one arena may not work in another. Entertaining and provocative, it will change how you view the world.

[Ryan Greer · Rating 4 out of 5]

This book was excellent. I was intrigued at first, then skepticism crept in as I started to wonder if Michael Blastland was some sort of internet quack who was tamping down his theories on flat earth and birds being robots and whatnot to write a book for the "mainstream" but, it turns out, he seems to be a pretty thoughtful and careful journalist who thinks words are important, and recognizes that we all have a tendency to exaggerate, especially if it supports our particular cause.

All of that to say, I really enjoyed this one, and although it was easy to get lost in the weeds in several chapters, I think the overall point he is trying to make, that we could all do with a bit more humility when it comes to the things we think we know, is a really important one. The book is full of interesting anecdotes, coupled with reminders that interesting anecdotes are often misunderstood or taken out of context or generalized to account for other factors... but he does a good job of both presenting a compelling argument while encouraging folks to not be entirely fooled by compelling arguments, things are invariably more complicated than we are ready to admit.

[Shubrashankh Chatterjee · Rating 4 out of 5]

The Hidden Half is one of the few books out there which makes you extremely uncomfortable about the state of decision making in the world. This book should be a must-read for all decision-makers or for that matter anyone who is under the assumption that randomness is random and does not impact their lives or work outcomes. The book can be a nightmare for scientists/engineers/policymakers if they are not open to a humbling experience as it exposes the brittle knowledge paradox we all live with.
For anyone wanting to get the most out of this book, I would suggest take your time and think through what the author is stating about how there is a huge amount that we don't know and will probably never know about the world. This does not mean we should stop making decisions or acting. The only takeaway is that we should be more humbled and conservative with the believes that we live by or act with, as most of our decisions have a very high possibility of not turning up as we want them to be due to the huge number of hidden factors which are not under our control.

[Loay Leon Hady · Rating 5 out of 5]

Refreshingly open minded

Being in education, it’s hard to avoid speakers popping up with random ‘proof’ for some new thing based on some research.

When it then fails to materialise with benefits expected and it demotivates teachers, many assume themselves to blame, when nothing could be further from the truth.

This book reminds us that something happening somewhere is not an automatic call for copy and paste. Nor is it even an expectation for the same people to be even replicate what they previously did.

It looks at many aspects of society and tries to explain the lack of awareness and replication on many levels. ‘It’s not our GDP’ being one of my favourite.

I think it will bring more comfort to many than self help books as it gives you an insight to the beautifully boundaries of research and thus assumptions of knowledge.

Look forward to reading more by Mr. Blastland

[Dolf van der Haven · Rating 3 out of 5]

Kudos to the cover illustrator, who constituted half (pun intended) of the reason why I bought this book.
The premise of the subject matter of the book was the other half, namely, why is it that there is variation in outcomes even if circumstances are seemingly the same? The first chapter about genetically identical Marmorkrebs in identical fishtanks producing offspring that has significant variation illustrated this mystery very well.
The rest of the book went downhill with various anecdotal "explanations" that stayed awkwardly on the surface, talking about poor research methods, correlation not being causation, missed root causes, etc. Things so obvious that in the end it still remains a mystery why there is such uncertainty about outcomes. The mystery of the Marmorkrebs never gets solved either.
Pair this with a preference of the author for wordy, overcomplicated sentences and you get an informative, but hardly convincing book.
Points detracted for mentioning Brexit and Trump.