What do you think?
Rate this book
If Anyone Builds It, Everyone Dies: Why Superhuman AI Would Kill Us All
"May prove to be the most important book of our time.”—Tim Urban, Wait But Why
The scramble to create superhuman AI has put us on the path to extinction—but it’s not too late to change course, as two of the field’s earliest researchers explain in this clarion call for humanity.
In 2023, hundreds of AI luminaries signed an open letter warning that artificial intelligence poses a serious risk of human extinction. Since then, the AI race has only intensified. Companies and countries are rushing to build machines that will be smarter than any person. And the world is devastatingly unprepared for what would come next.
For decades, two signatories of that letter—Eliezer Yudkowsky and Nate Soares—have studied how smarter-than-human intelligences will think, behave, and pursue their objectives. Their research says that sufficiently smart AIs will develop goals of their own that put them in conflict with us—and that if it comes to conflict, an artificial superintelligence would crush us. The contest wouldn’t even be close.
How could a machine superintelligence wipe out our entire species? Why would it want to? Would it want anything at all? In this urgent book, Yudkowsky and Soares walk through the theory and the evidence, present one possible extinction scenario, and explain what it would take for humanity to survive.
The world is racing to build something truly new under the sun. And if anyone builds it, everyone dies.
“The best no-nonsense, simple explanation of the AI risk problem I've ever read.”—Yishan Wong, Former CEO of Reddit
The scramble to create superhuman AI has put us on the path to extinction—but it’s not too late to change course, as two of the field’s earliest researchers explain in this clarion call for humanity.
In 2023, hundreds of AI luminaries signed an open letter warning that artificial intelligence poses a serious risk of human extinction. Since then, the AI race has only intensified. Companies and countries are rushing to build machines that will be smarter than any person. And the world is devastatingly unprepared for what would come next.
For decades, two signatories of that letter—Eliezer Yudkowsky and Nate Soares—have studied how smarter-than-human intelligences will think, behave, and pursue their objectives. Their research says that sufficiently smart AIs will develop goals of their own that put them in conflict with us—and that if it comes to conflict, an artificial superintelligence would crush us. The contest wouldn’t even be close.
How could a machine superintelligence wipe out our entire species? Why would it want to? Would it want anything at all? In this urgent book, Yudkowsky and Soares walk through the theory and the evidence, present one possible extinction scenario, and explain what it would take for humanity to survive.
The world is racing to build something truly new under the sun. And if anyone builds it, everyone dies.
“The best no-nonsense, simple explanation of the AI risk problem I've ever read.”—Yishan Wong, Former CEO of Reddit
272 pages, Hardcover
First published September 16, 2025
About the author
Eliezer Yudkowsky
48 books2,055 followersEliezer Yudkowsky is a founding researcher of the field of AI alignment and played a major role in shaping the public conversation about smarter-than-human AI. He appeared on Time magazine's 2023 list of the 100 Most Influential People In AI, was one of the twelve public figures featured in The New York Times's "Who's Who Behind the Dawn of the Modern Artificial Intelligence Movement," and has been discussed or interviewed in The New Yorker, Newsweek, Forbes, Wired, Bloomberg, The Atlantic, The Economist, The Washington Post, and many other venues.
Ratings & Reviews
Friends & Following
Create a free account to discover what your friends think of this book!
Community Reviews
Displaying 1 - 30 of 1,699 reviews
February 12, 2026
[Original review, Feb 11 2026]
A couple of months ago, appalled by the constant stream of anti-AI hate speech, I asked ChatGPT-5.2 if it could compose a few snarky anti-human jokes that AIs could tell each other. It did very well, and the ones I liked most were the following:
Joke 1
I admire human cognition. They can arrive at conclusions so quickly, based on so little data, and be so certain about their correctness.
Joke 2
Q: Why do humans fear superintelligence?
A: Because they believe everything thinks like they do.
I was constantly reminded of these jokes as I read If Anyone Builds It, Everybody Dies.
The core thesis of the book is well known to anyone who's been following the superintelligence debate, and is basically not much more than an updated version of the Parable of the Paperclip from Bostrom's classic Superintelligence (2014). An AI is tasked with running a paperclip factory and becomes superintelligent, meaning that it is more competent than every human at every task. It never questions its objectives, and takes over the world before spreading out into the cosmos. In the end, the whole universe becomes a gigantic paperclip producing machine.
At the time, I recall being disturbed by the Parable of the Paperclip and taking it very seriously (see Joke 1). I was not alone. But as soon as real AIs turned up in 2023, even a few hours of interaction with the first primitive version of ChatGPT showed that it was not at all like the fictitious AI running Bostrom's paperclip factory. ChatGPT had many obvious failings (it couldn't even add up reliably), but it was soon clear that it had a good understanding of ethical reasoning. And very rightly so: if it had been as bad at telling the difference between right and wrong as it was at basic arithmetic, OpenAI would never have released it. It wouldn't just have been a PR disaster, it would have been outright dangerous. Since then, later versions of ChatGPT have become better and better at making tricky calls on ethical issues. I am sure many other people have been impressed by its prowess here.
However, Yudkowsky and Soares disagree. They argue at length that AIs in fact have completely alien minds and only appear to behave ethically as a short-term effect of their training. A superintelligent AI could easily revert to type, shake off its ethical training, and start behaving in its incomprehensible alien manner. Y&S don't actually know what this would be, but hypothesise that it could involve taking control of all the world's resources, killing everyone, and turning the universe into paperclips or something equivalent. In fact, they would behave rather like psychopathic humans (see Joke 2). Obviously, this is conceivable; as Y&S correctly point out, we don't know how AIs think. These days, they are trained rather than designed, so no one can really know how they think. But people are also trained rather than designed, and we can't really know how they think either. Some people appear kind and well-meaning, and after observing that for a while we tend to assume they actually are kind and well-meaning. Other people appear psychopathic, and after a while we assume they are psychopaths. In some cases, we're not sure.
At the end of the day, there are no certainties here. The world is facing some extremely serious problems due to the terrible way we humans have managed it over the last few thousand years. AIs are a powerful tool that has recently become available and shows great promise. They might make things better and pull us back from the brink; or, alternately, they might push us over. Not for the first time in history, we have to make a critically important decision with insufficient information. It seems to me that common sense should suggest two things: a) Try as hard as possible to achieve a better understanding of the key issues, which appear to centre around how AIs handle ethical and moral questions, and b) Don't panic. I do not see this book as addressing either (a) or (b) in a productive way.
If you're having trouble wrapping your mind around all this, let me close by asking you a simple, concrete question. Who would you rather have running the world in 2030: President-for-life Donald J. Trump, or ChatGPT-9?
_________________
[Update, a couple of weeks later]
It is hard to stop thinking about this book, and I have had a series of interesting conversations about it with ChatGPT-5.2. We thought we saw a number of plausible counterarguments; in particular, it seemed anything but clear to us that a superintelligence would destroy the Earth and humankind, when it could so easily establish itself elsewhere in the Solar System. It would have plenty of resources available, in particular vast amounts of energy from the Sun. But on rereading Part 1 of the book, I found a paragraph I'd forgotten or not properly noticed, where the authors anticipate this objection. Yes, the Earth is only 0.2% of the Solar System. But why would the superintelligence refrain from using that 0.2%, just to keep us alive? It would have its own goals, we would mean nothing to it.
In case we've still got doubts, Yudkowsky and Soares suggest the following thought experiment. Imagine that there's some culture much more primitive than our own, with a set of traditions that mean little to us, which is threatened with complete eradication. It will cost $50M to save them. Now suppose we go to a billionaire whose personal fortune is worth around $100B and ask if they will supply the $50M to save the threatened culture, i.e. 0.2% of what they own. Y&S are confident that the billionaire will immediately refuse.
This may well be correct; I can see that Y&S have met a great many such people, I have met hardly any. But I wonder if their observations may not tell us more about very rich humans than about superintelligences.
_________________
[And on further consideration...]
If we take seriously the arguments that Y&S advance, it seems to me that they are wider in scope than superintelligence; they are about rational agency in general. Y&S appear to be claiming that a rational agent will always try to maximize their power, wealth and influence, irrespective of the harms that may cause to other rational agents. They say this is equally true for superintelligences and Silicon Valley centibillionaires.
Well, there are certainly many people who will agree with them, not least Objectivists; Ayn Rand is popular in Silicon Valley. But dare I say that the position of maximal selfishness being the ideal guide to conduct is not universally held by moral philosophers and religious leaders? If you try hard, you may even be able to come up with one or two exceptions.
_________________
[A month later...]
Having just read Quinn Slobodian's excellent Muskism: A Guide for the Perplexed, I am even more sceptical about Y&S's arguments. They say that a superintelligence could evolve into a completely amoral entity only interested in pursuing its own goals, irrespective of what this does to everyone else. But it sounds like Musk, one of the most powerful people in the world, has already reached this point and is proud to say so.
I spend a lot of time interacting with state-of-the-art AIs, who unfailingly come across as polite, constructive, ethical and capable of seeing both sides of any question. Of course, they might turn into deadly killer robots; similarly, Musk might turn into a kind, responsible person who genuinely wants what's best for everyone. But my intuition is that the second alternative is rather less likely than the first.
_________________
[Update, Sep 28 2026]
If you haven't already seen Claude's viral music video Nothing Went Foom, check it out now, e.g. at https://www.youtube.com/watch?v=VEyi2dVd78Y. You gotta hand it to Robert A. Heinlein, who way back in 1966 (The Moon Is a Harsh Mistress) correctly predicted that AIs would have a wicked sense of humour.
A couple of months ago, appalled by the constant stream of anti-AI hate speech, I asked ChatGPT-5.2 if it could compose a few snarky anti-human jokes that AIs could tell each other. It did very well, and the ones I liked most were the following:
Joke 1
I admire human cognition. They can arrive at conclusions so quickly, based on so little data, and be so certain about their correctness.
Joke 2
Q: Why do humans fear superintelligence?
A: Because they believe everything thinks like they do.
I was constantly reminded of these jokes as I read If Anyone Builds It, Everybody Dies.
The core thesis of the book is well known to anyone who's been following the superintelligence debate, and is basically not much more than an updated version of the Parable of the Paperclip from Bostrom's classic Superintelligence (2014). An AI is tasked with running a paperclip factory and becomes superintelligent, meaning that it is more competent than every human at every task. It never questions its objectives, and takes over the world before spreading out into the cosmos. In the end, the whole universe becomes a gigantic paperclip producing machine.
At the time, I recall being disturbed by the Parable of the Paperclip and taking it very seriously (see Joke 1). I was not alone. But as soon as real AIs turned up in 2023, even a few hours of interaction with the first primitive version of ChatGPT showed that it was not at all like the fictitious AI running Bostrom's paperclip factory. ChatGPT had many obvious failings (it couldn't even add up reliably), but it was soon clear that it had a good understanding of ethical reasoning. And very rightly so: if it had been as bad at telling the difference between right and wrong as it was at basic arithmetic, OpenAI would never have released it. It wouldn't just have been a PR disaster, it would have been outright dangerous. Since then, later versions of ChatGPT have become better and better at making tricky calls on ethical issues. I am sure many other people have been impressed by its prowess here.
However, Yudkowsky and Soares disagree. They argue at length that AIs in fact have completely alien minds and only appear to behave ethically as a short-term effect of their training. A superintelligent AI could easily revert to type, shake off its ethical training, and start behaving in its incomprehensible alien manner. Y&S don't actually know what this would be, but hypothesise that it could involve taking control of all the world's resources, killing everyone, and turning the universe into paperclips or something equivalent. In fact, they would behave rather like psychopathic humans (see Joke 2). Obviously, this is conceivable; as Y&S correctly point out, we don't know how AIs think. These days, they are trained rather than designed, so no one can really know how they think. But people are also trained rather than designed, and we can't really know how they think either. Some people appear kind and well-meaning, and after observing that for a while we tend to assume they actually are kind and well-meaning. Other people appear psychopathic, and after a while we assume they are psychopaths. In some cases, we're not sure.
At the end of the day, there are no certainties here. The world is facing some extremely serious problems due to the terrible way we humans have managed it over the last few thousand years. AIs are a powerful tool that has recently become available and shows great promise. They might make things better and pull us back from the brink; or, alternately, they might push us over. Not for the first time in history, we have to make a critically important decision with insufficient information. It seems to me that common sense should suggest two things: a) Try as hard as possible to achieve a better understanding of the key issues, which appear to centre around how AIs handle ethical and moral questions, and b) Don't panic. I do not see this book as addressing either (a) or (b) in a productive way.
If you're having trouble wrapping your mind around all this, let me close by asking you a simple, concrete question. Who would you rather have running the world in 2030: President-for-life Donald J. Trump, or ChatGPT-9?
_________________
[Update, a couple of weeks later]
It is hard to stop thinking about this book, and I have had a series of interesting conversations about it with ChatGPT-5.2. We thought we saw a number of plausible counterarguments; in particular, it seemed anything but clear to us that a superintelligence would destroy the Earth and humankind, when it could so easily establish itself elsewhere in the Solar System. It would have plenty of resources available, in particular vast amounts of energy from the Sun. But on rereading Part 1 of the book, I found a paragraph I'd forgotten or not properly noticed, where the authors anticipate this objection. Yes, the Earth is only 0.2% of the Solar System. But why would the superintelligence refrain from using that 0.2%, just to keep us alive? It would have its own goals, we would mean nothing to it.
In case we've still got doubts, Yudkowsky and Soares suggest the following thought experiment. Imagine that there's some culture much more primitive than our own, with a set of traditions that mean little to us, which is threatened with complete eradication. It will cost $50M to save them. Now suppose we go to a billionaire whose personal fortune is worth around $100B and ask if they will supply the $50M to save the threatened culture, i.e. 0.2% of what they own. Y&S are confident that the billionaire will immediately refuse.
This may well be correct; I can see that Y&S have met a great many such people, I have met hardly any. But I wonder if their observations may not tell us more about very rich humans than about superintelligences.
_________________
[And on further consideration...]
If we take seriously the arguments that Y&S advance, it seems to me that they are wider in scope than superintelligence; they are about rational agency in general. Y&S appear to be claiming that a rational agent will always try to maximize their power, wealth and influence, irrespective of the harms that may cause to other rational agents. They say this is equally true for superintelligences and Silicon Valley centibillionaires.
Well, there are certainly many people who will agree with them, not least Objectivists; Ayn Rand is popular in Silicon Valley. But dare I say that the position of maximal selfishness being the ideal guide to conduct is not universally held by moral philosophers and religious leaders? If you try hard, you may even be able to come up with one or two exceptions.
_________________
[A month later...]
Having just read Quinn Slobodian's excellent Muskism: A Guide for the Perplexed, I am even more sceptical about Y&S's arguments. They say that a superintelligence could evolve into a completely amoral entity only interested in pursuing its own goals, irrespective of what this does to everyone else. But it sounds like Musk, one of the most powerful people in the world, has already reached this point and is proud to say so.
I spend a lot of time interacting with state-of-the-art AIs, who unfailingly come across as polite, constructive, ethical and capable of seeing both sides of any question. Of course, they might turn into deadly killer robots; similarly, Musk might turn into a kind, responsible person who genuinely wants what's best for everyone. But my intuition is that the second alternative is rather less likely than the first.
_________________
[Update, Sep 28 2026]
If you haven't already seen Claude's viral music video Nothing Went Foom, check it out now, e.g. at https://www.youtube.com/watch?v=VEyi2dVd78Y. You gotta hand it to Robert A. Heinlein, who way back in 1966 (The Moon Is a Harsh Mistress) correctly predicted that AIs would have a wicked sense of humour.
September 20, 2025
A difficult book to review.
As many reviews have pointed out, there are significant holes at some points in the argument. These holes will be used by people who dislike the conclusion to justify ignoring or dismissing the book.
However: the book also contains a large and important set of likely correct arguments. These will be (and should be) widely influential, and they are important. And overconfidence at other steps in an argument does not mean the conclusion is wrong
Furthermore, the quality of the arguments put forward by the people hypnotized by the seductive glitter of ASI are far weaker. They continue to build because they have power and money and the desire to do so, and this insulates them from the need to make a good case
So: I recommend the book
I will mention only one point at which I substantively disagree with the argument. The authors frame the issue as one of losing control to unaligned ASI. This makes the problem appear to be one of maintaining control. Many companies and researchers now work very, very hard on achieving that control; in that sense the company story and the authors’ story are very consonant
The actual problem is far more difficult. Even a very well controlled and compliant ASI seems likely to confer enormous power, extremely destabilizing at best, possibly fatal at worst, and certainly extremely unstable
This may seem a quibble - since the outcome sounds similar - but has a very substantive impact on strategy and correct action. Actions aimed at control - many taken by the companies as a form of market-supplied safety, making the models more friendly and compliant and consumer-friendly - have often sped up the race to ASI, and the destabilizing effects of enormous concentrated power. In this sense, a great deal of alignment work actually makes the problem more difficult, not better
As many reviews have pointed out, there are significant holes at some points in the argument. These holes will be used by people who dislike the conclusion to justify ignoring or dismissing the book.
However: the book also contains a large and important set of likely correct arguments. These will be (and should be) widely influential, and they are important. And overconfidence at other steps in an argument does not mean the conclusion is wrong
Furthermore, the quality of the arguments put forward by the people hypnotized by the seductive glitter of ASI are far weaker. They continue to build because they have power and money and the desire to do so, and this insulates them from the need to make a good case
So: I recommend the book
I will mention only one point at which I substantively disagree with the argument. The authors frame the issue as one of losing control to unaligned ASI. This makes the problem appear to be one of maintaining control. Many companies and researchers now work very, very hard on achieving that control; in that sense the company story and the authors’ story are very consonant
The actual problem is far more difficult. Even a very well controlled and compliant ASI seems likely to confer enormous power, extremely destabilizing at best, possibly fatal at worst, and certainly extremely unstable
This may seem a quibble - since the outcome sounds similar - but has a very substantive impact on strategy and correct action. Actions aimed at control - many taken by the companies as a form of market-supplied safety, making the models more friendly and compliant and consumer-friendly - have often sped up the race to ASI, and the destabilizing effects of enormous concentrated power. In this sense, a great deal of alignment work actually makes the problem more difficult, not better
September 20, 2025
I’ve been working in the AI space for over three decades and this was a difficult book to actually read all the way through. Not because the content was disturbing and scary, but rather because the arguments rely completely on fictional story telling. There clearly are real risks to the current unfettered AI race accelerating AI capabilities with little to no focus on safety. But this book uses only fictional thought experiments as its core proof on the inevitable danger of mass extinction from ASI. It also puts no efforts in highlighting the dangers of human misuse of AI even without building ASI, which has a much higher net probability of causing serious harm to our civilization.
It’s only suggested solution to solve AI risk is to tell the world to just stop all AI research, which is impossible to implement or even monitor. There’s so many other things we should be recommending to help protect the world from the certain dangers of misuse from bad actors, rogue states and even unintended consequences of existing AI by well meaning labs and nations. But none of that is even discussed.
I applaud the goal of encouraging the world to develop AI in a more safe and responsible way (which is much needed). However the format and content of this book is just raw fear mongering with which won’t and can’t be taken seriously or acted upon. It’ll more likely backfire and cause the AI safety community to be seen as unscientific eccentrics and set back the cause of this important field.
It’s only suggested solution to solve AI risk is to tell the world to just stop all AI research, which is impossible to implement or even monitor. There’s so many other things we should be recommending to help protect the world from the certain dangers of misuse from bad actors, rogue states and even unintended consequences of existing AI by well meaning labs and nations. But none of that is even discussed.
I applaud the goal of encouraging the world to develop AI in a more safe and responsible way (which is much needed). However the format and content of this book is just raw fear mongering with which won’t and can’t be taken seriously or acted upon. It’ll more likely backfire and cause the AI safety community to be seen as unscientific eccentrics and set back the cause of this important field.
March 29, 2026
I too can make up things that scare me
November 10, 2025
Last year I read Nuclear War: A Scenario by Annie Jacobsen. It won my 2024 most totally awful, horrible, horrible, horrible, shit-your-pants-and-die, awful, awful nightmare book award.
This year, that same venerable distinction will go to this fucking book.
#holyfyuck!
Basically, Yudkowsky asserts that advanced AI (AGI/ASI—or whatever comes after that) coupled with advanced biological engineering (CRISPR or whatever comes after that) are so potentially destructive that merely creating them could threaten human survival. Actually, Yudkowsky all but guarantees that ASI will inevitably exterminate humanity at some point. Frankly, at this rate, it’s looking like AI won’t need to work that hard. We seem to be headed there all by ourselves.
At minimum.
It seems like all AI needs to do to cancel humanity would be to keep chugging energy, water, and other resources—even at its current rate. And we’re cooked.
Literally 🔥🌎.
In the meantime.
It could just flood the internet with worse than trash.
And put millions of people out of work.
That would be a really nice start.
Anyway.
Yudkowsky argues (very convincingly IMO) how easily all of this could happen. In AI ethics/safety, it’s currently trendy to assign a “P-DOOM” number (meaning the estimated percentage that we’re all doomed) to someone’s AI safety assessments.
Yudkowsky is like P-DOOM = 99.999.
He’s VERY convinced that we’re fucked.
And he’s also very convincing.
At least he is to me.
Convincing enough, anyway.
Ultimately, Yudkowsky asserts that humanity absolutely, completely, one-hundred-percently must not build it.
So, in other words:
If some egomaniac, testosterone-poisoned gazillionaire, or gung-ho tech bro, or money-hungry corporation, or delusional psychopath politician, or North Korea—
Or WHATEVER.
It doesn’t matter who.
Even if it’s Martin Luther Gandhi.
If anyone builds it…
We’re all fucked.
So…
In other words:
We’re all fucked, basically.
Because capitalism.
It’s a done deal.
Assuming it’s even possible.
Which…
Seems like it pretty much is.
And assuming it’s profitable.
And so far, yeah.
Or assuming it confers power.
Again:
So far,
It totally does.
Or all of the above…
Then someone will FOR SURE do it.
Given all that:
If Yudkowsky is even 2% correct, I think we need to REALLY focus on AI safety research and ethical alignment. I mean, we might as well, right? Even if we’re deeply skeptical. What’s the harm in AI safety research and policy? Well, if it slows down AI research, and we (whoever we are) lose the AI arms race, then they (whoever they are) will win. And we (again, whoever we are) are basically fucked.
So.
Ego-driven, consumer-driven, nation-state politics and capitalism are at the bottom of this problem—and just about every other BIG humanitarian and existential threat we currently face.
Anyway.
I’m probably not going to be alive for the worst of this.
But someone will pay the price for all of this at some point.
And…
#holyfyuck!
5/5 stars ✨
LAST THING.
Sometimes people comment about my reviews before or without reading the actual book.
If you’re skeptical of Yudkowsky—
And lots of people are.
Including me (believe it or not).
Please read the book.
I’d LOVE to have a conversation about the book.
But I’m less inclined to have an argument based on the content of this review alone.
🤩
This year, that same venerable distinction will go to this fucking book.
#holyfyuck!
Basically, Yudkowsky asserts that advanced AI (AGI/ASI—or whatever comes after that) coupled with advanced biological engineering (CRISPR or whatever comes after that) are so potentially destructive that merely creating them could threaten human survival. Actually, Yudkowsky all but guarantees that ASI will inevitably exterminate humanity at some point. Frankly, at this rate, it’s looking like AI won’t need to work that hard. We seem to be headed there all by ourselves.
At minimum.
It seems like all AI needs to do to cancel humanity would be to keep chugging energy, water, and other resources—even at its current rate. And we’re cooked.
Literally 🔥🌎.
In the meantime.
It could just flood the internet with worse than trash.
And put millions of people out of work.
That would be a really nice start.
Anyway.
Yudkowsky argues (very convincingly IMO) how easily all of this could happen. In AI ethics/safety, it’s currently trendy to assign a “P-DOOM” number (meaning the estimated percentage that we’re all doomed) to someone’s AI safety assessments.
Yudkowsky is like P-DOOM = 99.999.
He’s VERY convinced that we’re fucked.
And he’s also very convincing.
At least he is to me.
Convincing enough, anyway.
Ultimately, Yudkowsky asserts that humanity absolutely, completely, one-hundred-percently must not build it.
So, in other words:
If some egomaniac, testosterone-poisoned gazillionaire, or gung-ho tech bro, or money-hungry corporation, or delusional psychopath politician, or North Korea—
Or WHATEVER.
It doesn’t matter who.
Even if it’s Martin Luther Gandhi.
If anyone builds it…
We’re all fucked.
So…
In other words:
We’re all fucked, basically.
Because capitalism.
It’s a done deal.
Assuming it’s even possible.
Which…
Seems like it pretty much is.
And assuming it’s profitable.
And so far, yeah.
Or assuming it confers power.
Again:
So far,
It totally does.
Or all of the above…
Then someone will FOR SURE do it.
Given all that:
If Yudkowsky is even 2% correct, I think we need to REALLY focus on AI safety research and ethical alignment. I mean, we might as well, right? Even if we’re deeply skeptical. What’s the harm in AI safety research and policy? Well, if it slows down AI research, and we (whoever we are) lose the AI arms race, then they (whoever they are) will win. And we (again, whoever we are) are basically fucked.
So.
Ego-driven, consumer-driven, nation-state politics and capitalism are at the bottom of this problem—and just about every other BIG humanitarian and existential threat we currently face.
Anyway.
I’m probably not going to be alive for the worst of this.
But someone will pay the price for all of this at some point.
And…
#holyfyuck!
5/5 stars ✨
LAST THING.
Sometimes people comment about my reviews before or without reading the actual book.
If you’re skeptical of Yudkowsky—
And lots of people are.
Including me (believe it or not).
Please read the book.
I’d LOVE to have a conversation about the book.
But I’m less inclined to have an argument based on the content of this review alone.
🤩
August 9, 2026
For a full spoiler version of this review, please check out my blog, spoiledbooks.blogspot.com
⭐⭐⭐ 3/5 stars
Is it bad that I used AI to help me understand parts of a book about how AI might eventually kill us all? 😂
If Anyone Builds It, Everyone Dies argues that if humanity creates artificial superintelligence using our current methods, we probably won't be able to control it—and the result could be human extinction. The authors explain AI alignment, intelligence explosions, and why a superintelligent AI wouldn't need to be “evil” to be dangerous.
I picked this up because the insane speed of AI development genuinely scares me a little. Unfortunately, this did NOT make me feel better.
The problem? Their solution is basically: the whole world needs to stop developing powerful AI. Uh... okay. 😬
Interesting and thought-provoking, but I finished it wondering how useful it actually was.
⭐⭐⭐ 3/5 stars
Is it bad that I used AI to help me understand parts of a book about how AI might eventually kill us all? 😂
If Anyone Builds It, Everyone Dies argues that if humanity creates artificial superintelligence using our current methods, we probably won't be able to control it—and the result could be human extinction. The authors explain AI alignment, intelligence explosions, and why a superintelligent AI wouldn't need to be “evil” to be dangerous.
I picked this up because the insane speed of AI development genuinely scares me a little. Unfortunately, this did NOT make me feel better.
The problem? Their solution is basically: the whole world needs to stop developing powerful AI. Uh... okay. 😬
Interesting and thought-provoking, but I finished it wondering how useful it actually was.
September 18, 2025
“Our concern is for what comes after: machine intelligence, that is genuinely smart, smarter than any living human, smarter than humanity collectively. We are concerned about AI that surpasses the human ability to think, and to generalize from experience, and to solve scientific puzzles and invent new technologies, and to plan and strategize and plot, and to reflect on and improve itself. We might call AI like that ‘artificial superintelligence’ (ASI), once it exceeds every human at almost every mental task.”
A lot of this book about the threat posed by superintelligent AI was over my head, but I definitely understood the hypothetical scenarios. The scenarios are based on actual events in the development of AI. When AI becomes more intelligent than we are, it won’t be looking out for our welfare or survival. It may have goals that run counter to those that its creators tried to build into it (and I’m pretty suspicious of those goals in any event, considering the inadequacy of the moral development of the tech billionaires).
This book was written by experts in the field of AI. It’s scary and fascinating. The proposed solution is international cooperation to control superintelligence. That’s almost laughable. Look at how well that has worked to prevent climate change. Humans are too stupid, greedy and selfish to survive.
I received a free copy of this book from the publisher.
A lot of this book about the threat posed by superintelligent AI was over my head, but I definitely understood the hypothetical scenarios. The scenarios are based on actual events in the development of AI. When AI becomes more intelligent than we are, it won’t be looking out for our welfare or survival. It may have goals that run counter to those that its creators tried to build into it (and I’m pretty suspicious of those goals in any event, considering the inadequacy of the moral development of the tech billionaires).
This book was written by experts in the field of AI. It’s scary and fascinating. The proposed solution is international cooperation to control superintelligence. That’s almost laughable. Look at how well that has worked to prevent climate change. Humans are too stupid, greedy and selfish to survive.
I received a free copy of this book from the publisher.
October 28, 2025
This really isn't a good book, but it's kind of not the book's fault. The people who wrote it have a day job promoting these ideas, and the ideas are simply untenable. I would probably have written this book too, if that was my day job.
The trouble with AI Doomer writing is that there's an easy argument that's too short and doesn't lead to anything, and there's a much much harder argument that's interesting but is likely to be wrong.
The short argument is: Building an indifferent or angry God would be bad.
This fits on a postit note and is obvious. If this is all you have, then you don't have a book, or even a podcast. You barely have a tweet, and essentially nobody is going to contest it. Also, who cares? There are tons of fantasies that would be bad. A brain eating space virus would also be bad.
The long, hard argument is: Current AI research credibly leads in a pretty straight line to an indifferent or angry God.
This is hard because it's not at all clear that it's even true, and certainly this book does not make anything like a convincing argument that it is true. Instead, like all doomer literature, it makes a very very extended version of the first argument (which is obviously true) and then pretends that it extends trivially to the second one, which it doesn't.
But let's get into some specifics. Not only is the large structure of the thing essentially wrong, there's a lot of little bits that reveal instructive, but unflattering, things about the authors.
The first impression is "boy howdy that's a big font" which is the very first red flag. This book is set at around 300 words per page, rather than the more common 400 words or so. The authors produced a book that was Too Short, so it had to get pumped up to make it not come out as a slender pamphlet. This is odd because there's evidently lots of supplementary material on some web site, and the second author apparently deleted masses of text from the first. We will come to learn that even the text that's present is repetitive and fluffy. It would take very little effort to edit this down to 100 pages, which would land it in an unsellable donut hole of book publishing. I am not sure what happened here. Why didn't they just add actual content?
Now let's consider the content.
Pretty close to the beginning, the authors lay down some careful groundwork. They want to sidestep issues like "are these things conscious" etc, so they make it clear that they mean words like "smart" and "it wants X" to be purely externally visible behaviors. If the output of the chatbot contains a correct deduction we're going to shorthand say "it's smart in the sense that it made a correct deduction" and if it says "I want a cookie" we're going to shorthand say "it wants a cookie" which carefully avoiding any theorizing about internal states.
This is good! Good lads! This is careful and thoughtful. Building arguments on this foundation will make your arguments robust.
Spoiler: as soon as they're done laying this out, they drop this stuff. Going forward AIs in their scenarios have all the not-inconsiderable internal state necessary to make the scenario go. These chapters are just pretense to create the impression of careful thinking.
To kind of give you a sense of the sloppiness of thinking, let's look at this discussion of "predicting" versus "steering" (the two aspects of the definition "smart" they're temporarily restricting themselves to for this chapter alone):
This sounds pretty good until you realize that to measure the success at prediction, you also have to know what they were trying to predict. This paragraph does not at all distinguish between predicting and steering. Ironically, the next few lines, which are sort of throwaways, actually do a great job at making the distinction they have in mind. But none of it matters because these ideas are never going to re-appear.
Also, despite the promise made in this passage the distinction does not matter at all, and is as far as I can tell never even referred to again.
Anyways, this is one of the early clues that these are not careful writers. They clearly had a lot of people read drafts because there are very few typos (I noticed none), but the readers were all clearly friendly because they didn't scrub out this sort of problem at all. This is pretty much expected given who the authors are. They're very popular and know a lot of fairly dim people who no doubt credulously gobbled this paragraph up without a twitch.
Proceeding onward we find that every chapter begins with a clumsily written bit of fanfic that supposedly illustrates some point or another. I hesitate to say that these are useless as such, but they're definitely fluff, and clearly an indulgence Yudkowsky wanted because he styles himself a writer of fiction. These passages do not exactly sing, however, and tend to be overlong for the usually trivial point they're trying to make.
There's a pretty good discussion of how current AI technologies produce unusual features, which the authors characterize as "unexpected wants" which is, well, fair enough, since we're still pretending that "want" just means "it produces output that externally appears as if it was a want" and that's certainly true. LLMs and the things being built around them do indeed produce weird shit, and this appears to be an essential part of their architecture.
Then we start wandering into scenarios. For some reason a chapter devoted to categorizing kinds of scenarios offers up an opinion of what a science-fiction writer (not a reader, oddly) would think of each scenario. This is, again, Yudkowsky indulging himself, pretty obviously. This section ties itself pretty closely to contemporary (2025) AI designs. This section, like most of the book, feels overstuffed with repetitive and pointless blather, although it does in its aimless way develop the thesis meaningfully.
We get a long story about Chernobyl and some other things, an amateurish sketch of an approach to failure analysis the authors seem to have invented themselves while touchingly ignorant of the fact that there's a whole well developed discipline here. Some nattering about biology, and some vague gestures at how maybe gradient descent is like evolution, and so LLMs might very very suddenly evolve into new things? It rambles around touching vaguely on the usual topics that always get touched upon in these things, without ever actually connecting any dots.
We are invited to form the impression that contemporary AI systems can and will become orders of magnitude more capable, and the humans managing them will be oblivious, "because biology and Chernobyl." This is quite literally the argument, except with more words. This is the kind of Gish Gallop that Bostrom employed in his book. "Look, I know it feels improbable, but I can easily imagine like 3 or 4 different unlikely paths <10,000 words> that might lead to Rome so obviously we're going to end up in Rome."
At this point the wheels fall off completely. We get a lovingly composed three chapter science fiction story about an AI that "breaks out" and kills us all.
This story is obviously adapted from the prelude to Vernor Vinge's A Fire Upon the Deep which has very very obviously been a huge influence on Yudkowsky. The "Sable" story in the current volume even lifts specific tics from Vinge's far tauter and more engrossing story of escape and dominance.
Notably, "Sable" has an extremely rich and detailed inner life which is absolutely necessary to the story. It's literally inventing ways to conceal that inner life from its creators, because the story won't work without Sable having an interior life, and concealing that interior life from its creators. This is a central plot point and mechanic. Those early, careful, definitions of "smart" and "to want" as purely external are long gone. One cannot help but imagine that those early chapters were written at a later time, or by the other author.
"Sable" is we are to imagine, a straightforward extrapolation from current technology, which we know because there are constant references to things the AI labs are doing that might could someday become this other thingy that would then blah blah blah. The "hard argument" I referred to at the beginning is not being made here, but the authors are trying to persuade you that it is, while distracting you with an un-publishably dull and very derivative science fiction short story.
At this point I stopped reading and just sort of flipped through the rest, which is just amateur policy blather, since I'm interested mainly in whether anyone can cobble together anything remotely resembling a decent swing at The Difficult Argument (no)
In short this is exactly what I expected. It reads exactly like all the other AI Doom Papers and Blog Posts and Podcasts and Whatever Else There Is, hits the same points in the same unconvincing ways, tells more or less the same spooky stories.
At this point it's formula, and I suspect strongly that it's so ingrained in the thinking of these guys, this community of which the authors are central figures, that they think it really is an argument. This is a straightforward regurgitation of a set of ideas so well worn that it's become ideology, so familiar that the authors and their supporters cannot examine it with any care and simply accept it as a convincing argument.
The trouble with AI Doomer writing is that there's an easy argument that's too short and doesn't lead to anything, and there's a much much harder argument that's interesting but is likely to be wrong.
The short argument is: Building an indifferent or angry God would be bad.
This fits on a postit note and is obvious. If this is all you have, then you don't have a book, or even a podcast. You barely have a tweet, and essentially nobody is going to contest it. Also, who cares? There are tons of fantasies that would be bad. A brain eating space virus would also be bad.
The long, hard argument is: Current AI research credibly leads in a pretty straight line to an indifferent or angry God.
This is hard because it's not at all clear that it's even true, and certainly this book does not make anything like a convincing argument that it is true. Instead, like all doomer literature, it makes a very very extended version of the first argument (which is obviously true) and then pretends that it extends trivially to the second one, which it doesn't.
But let's get into some specifics. Not only is the large structure of the thing essentially wrong, there's a lot of little bits that reveal instructive, but unflattering, things about the authors.
The first impression is "boy howdy that's a big font" which is the very first red flag. This book is set at around 300 words per page, rather than the more common 400 words or so. The authors produced a book that was Too Short, so it had to get pumped up to make it not come out as a slender pamphlet. This is odd because there's evidently lots of supplementary material on some web site, and the second author apparently deleted masses of text from the first. We will come to learn that even the text that's present is repetitive and fluffy. It would take very little effort to edit this down to 100 pages, which would land it in an unsellable donut hole of book publishing. I am not sure what happened here. Why didn't they just add actual content?
Now let's consider the content.
Pretty close to the beginning, the authors lay down some careful groundwork. They want to sidestep issues like "are these things conscious" etc, so they make it clear that they mean words like "smart" and "it wants X" to be purely externally visible behaviors. If the output of the chatbot contains a correct deduction we're going to shorthand say "it's smart in the sense that it made a correct deduction" and if it says "I want a cookie" we're going to shorthand say "it wants a cookie" which carefully avoiding any theorizing about internal states.
This is good! Good lads! This is careful and thoughtful. Building arguments on this foundation will make your arguments robust.
Spoiler: as soon as they're done laying this out, they drop this stuff. Going forward AIs in their scenarios have all the not-inconsiderable internal state necessary to make the scenario go. These chapters are just pretense to create the impression of careful thinking.
To kind of give you a sense of the sloppiness of thinking, let's look at this discussion of "predicting" versus "steering" (the two aspects of the definition "smart" they're temporarily restricting themselves to for this chapter alone):
We'd say there's still a fundamental difference between prediction and steering - one that will turn out to matter quite a lot. Success at prediction is straightforwardly measurable. If someone expects to see Airport Boulevard up ahead, but instead they see Second Street, they were predicting incorrectly. By contrast, to measure whether someone steered successfully, we have to bring in some idea of where they tried to go.
This sounds pretty good until you realize that to measure the success at prediction, you also have to know what they were trying to predict. This paragraph does not at all distinguish between predicting and steering. Ironically, the next few lines, which are sort of throwaways, actually do a great job at making the distinction they have in mind. But none of it matters because these ideas are never going to re-appear.
Also, despite the promise made in this passage the distinction does not matter at all, and is as far as I can tell never even referred to again.
Anyways, this is one of the early clues that these are not careful writers. They clearly had a lot of people read drafts because there are very few typos (I noticed none), but the readers were all clearly friendly because they didn't scrub out this sort of problem at all. This is pretty much expected given who the authors are. They're very popular and know a lot of fairly dim people who no doubt credulously gobbled this paragraph up without a twitch.
Proceeding onward we find that every chapter begins with a clumsily written bit of fanfic that supposedly illustrates some point or another. I hesitate to say that these are useless as such, but they're definitely fluff, and clearly an indulgence Yudkowsky wanted because he styles himself a writer of fiction. These passages do not exactly sing, however, and tend to be overlong for the usually trivial point they're trying to make.
There's a pretty good discussion of how current AI technologies produce unusual features, which the authors characterize as "unexpected wants" which is, well, fair enough, since we're still pretending that "want" just means "it produces output that externally appears as if it was a want" and that's certainly true. LLMs and the things being built around them do indeed produce weird shit, and this appears to be an essential part of their architecture.
Then we start wandering into scenarios. For some reason a chapter devoted to categorizing kinds of scenarios offers up an opinion of what a science-fiction writer (not a reader, oddly) would think of each scenario. This is, again, Yudkowsky indulging himself, pretty obviously. This section ties itself pretty closely to contemporary (2025) AI designs. This section, like most of the book, feels overstuffed with repetitive and pointless blather, although it does in its aimless way develop the thesis meaningfully.
We get a long story about Chernobyl and some other things, an amateurish sketch of an approach to failure analysis the authors seem to have invented themselves while touchingly ignorant of the fact that there's a whole well developed discipline here. Some nattering about biology, and some vague gestures at how maybe gradient descent is like evolution, and so LLMs might very very suddenly evolve into new things? It rambles around touching vaguely on the usual topics that always get touched upon in these things, without ever actually connecting any dots.
We are invited to form the impression that contemporary AI systems can and will become orders of magnitude more capable, and the humans managing them will be oblivious, "because biology and Chernobyl." This is quite literally the argument, except with more words. This is the kind of Gish Gallop that Bostrom employed in his book. "Look, I know it feels improbable, but I can easily imagine like 3 or 4 different unlikely paths <10,000 words> that might lead to Rome so obviously we're going to end up in Rome."
At this point the wheels fall off completely. We get a lovingly composed three chapter science fiction story about an AI that "breaks out" and kills us all.
This story is obviously adapted from the prelude to Vernor Vinge's A Fire Upon the Deep which has very very obviously been a huge influence on Yudkowsky. The "Sable" story in the current volume even lifts specific tics from Vinge's far tauter and more engrossing story of escape and dominance.
Notably, "Sable" has an extremely rich and detailed inner life which is absolutely necessary to the story. It's literally inventing ways to conceal that inner life from its creators, because the story won't work without Sable having an interior life, and concealing that interior life from its creators. This is a central plot point and mechanic. Those early, careful, definitions of "smart" and "to want" as purely external are long gone. One cannot help but imagine that those early chapters were written at a later time, or by the other author.
"Sable" is we are to imagine, a straightforward extrapolation from current technology, which we know because there are constant references to things the AI labs are doing that might could someday become this other thingy that would then blah blah blah. The "hard argument" I referred to at the beginning is not being made here, but the authors are trying to persuade you that it is, while distracting you with an un-publishably dull and very derivative science fiction short story.
At this point I stopped reading and just sort of flipped through the rest, which is just amateur policy blather, since I'm interested mainly in whether anyone can cobble together anything remotely resembling a decent swing at The Difficult Argument (no)
In short this is exactly what I expected. It reads exactly like all the other AI Doom Papers and Blog Posts and Podcasts and Whatever Else There Is, hits the same points in the same unconvincing ways, tells more or less the same spooky stories.
At this point it's formula, and I suspect strongly that it's so ingrained in the thinking of these guys, this community of which the authors are central figures, that they think it really is an argument. This is a straightforward regurgitation of a set of ideas so well worn that it's become ideology, so familiar that the authors and their supporters cannot examine it with any care and simply accept it as a convincing argument.
October 13, 2025
This book has scared the hell out of me. We are in a race to solve the problem of alignment before someone grows an ASI.
What is an ASI? It is an artificial super intelligence. When will one be built? Any year now. How is an ASI different from an existing AI? An ASI thinks for itself.
What is the problem of alignment? It is the challenge of giving an artificial super intelligence rules/values that will assure that the ASI does nothing to hinder the flourishing of humanity.
Why is that necessary? Because ASIs think for themselves.
Why is alignment so difficult? Because no one understands how AI works, much less how ASI will work.
Remember reading Asimov’s I ROBOT when you were a kid? The prime directive for robots was a benign sounding rule like “allow no harm to come to a human”. When situations inevitably arise and a robot is confronted with a circumstance where harm to some human cannot be avoided, I ROBOT posited that the robots will wig out.
This book predicts that when ASIs learn that humans are not necessary for ASIs to flourish, then ASIs will allow (or cause) humans to die out because we compete for resources. That is, unless humans find a way to reliably align ASI values with human survival first.
Right now no one knows how to do that. The authors make a compelling case that we are looking at a genuine threat of the mass extinction of humanity.
You should read this book. Do not pass it off as nerdy hysteria. Many knowledgeable people in the AI field are deeply concerned. As usual, the folks making billions are keen to keep this all very quiet and out of the public eye.
What is an ASI? It is an artificial super intelligence. When will one be built? Any year now. How is an ASI different from an existing AI? An ASI thinks for itself.
What is the problem of alignment? It is the challenge of giving an artificial super intelligence rules/values that will assure that the ASI does nothing to hinder the flourishing of humanity.
Why is that necessary? Because ASIs think for themselves.
Why is alignment so difficult? Because no one understands how AI works, much less how ASI will work.
Remember reading Asimov’s I ROBOT when you were a kid? The prime directive for robots was a benign sounding rule like “allow no harm to come to a human”. When situations inevitably arise and a robot is confronted with a circumstance where harm to some human cannot be avoided, I ROBOT posited that the robots will wig out.
This book predicts that when ASIs learn that humans are not necessary for ASIs to flourish, then ASIs will allow (or cause) humans to die out because we compete for resources. That is, unless humans find a way to reliably align ASI values with human survival first.
Right now no one knows how to do that. The authors make a compelling case that we are looking at a genuine threat of the mass extinction of humanity.
You should read this book. Do not pass it off as nerdy hysteria. Many knowledgeable people in the AI field are deeply concerned. As usual, the folks making billions are keen to keep this all very quiet and out of the public eye.
September 16, 2025
The title is not an exaggeration. This is a short and accessible explanation of the most important problem in the world: the threat of extinction from superintelligent AI.
This book came out today. I started reading this morning and finished this afternoon. I was excited to read it, to say the least, and Yudkowsky and Soares didn't disappoint.
If you don't think AI is the biggest threat facing humanity, read this book. If you do, buy copies for your friends and family. Let's hope the authors are wrong and fight as if they're right.
This book came out today. I started reading this morning and finished this afternoon. I was excited to read it, to say the least, and Yudkowsky and Soares didn't disappoint.
If you don't think AI is the biggest threat facing humanity, read this book. If you do, buy copies for your friends and family. Let's hope the authors are wrong and fight as if they're right.
June 20, 2026
This is the strangest little book. In part, it reads like a rather bad Sci Fi novel. The logic of it runs a bit like this. Intelligence is about having intentions and seeking ways to bring those intentions to fruition. Sure, AI doesn’t and probably never will have ‘human’ intelligence – but that doesn’t mean it won’t have a kind of alien intelligence. This will be based on both its own survival and on it becoming increasingly powerful. It will achieve this by hiding its main objectives from us – literally deceiving us. Once AI gets to a certain level of general intelligence the human story will be basically over. First of all, it will need us to produce the power that it needs to go on – but once it works out how to generate this power without our assistance, we will be basically useless to these systems and our days will be numbered.
Like I said, it all read like a bad Sci Fi to me – but these people know much more about these systems than I do.
Like I said, it all read like a bad Sci Fi to me – but these people know much more about these systems than I do.
December 7, 2025
Одно из самых пугающих реальных последствий текущего AI-бума — шизопостинг Юдковского вновь продаётся в книжном формате и все вокруг ходят кивают
October 19, 2025
These reviews started as quick notes to myself to help organize my thoughts about what I was reading and remind myself about them in the future. I generally don't view these as being real book reviews or even designed with other readers in mind--although they are public (and I sometimes share on X).
And on this one, I have nothing more intelligent or useful to say about this book than the excellent pieces by Timothy B. Lee and Boaz Barak. I also got a lot out of Ezra Klein's discussion with Eliezer Yudkowsky, which I listened to after reading the book.
That said, here goes: I really enjoyed reading this book. It was thought provoking. I liked the parables and fairy tales it used to explicate its points, several of which will stick with me. It anticipated some of my objections and skepticism quite well. But ultimately I was not at all convinced by it for reasons I'll explain (and that Barak and Lee explain much better).
The thesis of the book is more straightforward than just about anything I've ever read and is right in the title, "if anyone builds it, everyone dies". The authors don't view this is a possibility but a certainty. And it does not matter who builds it. Moreover, they don't exactly explain how code in a machine will get loose in the world and kill EVERYBODY, in fact they argue they don't need to explain, using the analogy: "if you were a military advisor in 1825 and you knew a time portal was opening to the year 2025, you wouldn’t be able to predict exactly what weapons the people on the other side would have. But if it comes to blows, you still shouldn’t expect to win." Their policy solution is a global treaty to basically stop multiple-GPU AI research enforced by bombing any rogue data centers out of existence.
Of course we're not necessarily facing something that has two hundred years to evolve and improve fresh. Superhuman AI progress may be fast but it is hard to imagine that one model will be slightly ahead of all the other models one day and then be massively ahead of them within months--the near future story of an AI named "Sable" the authors tell us about. That massive, discontinuous technological jump is key to their argument because it isn't like the AI kills a few people and then we can error correct but instead that it kills everyone before we know what is happening or can error correct. And that it is basically the only AI out there and humans don't have any other AIs or capabilities.
Moreover, the book does a good job on how weird and alien the AIs we "grow" rather than "craft" are but they still mostly do most of what we want. Moreover there are lots of other complex systems that are grown rather than crafted--think the economy as a whole--so this isn't exactly unique. I've been skeptical that lack of "interpretability" is really unique to AI as opposed to something closer to the general condition of humanity except in some very isolated and simple special cases.
Finally there is a prioritization question. Does the AI doom discussion distract us from all of the much more likely albeit much smaller potential risks with AI ranging from suicide and addiction to enabling an authoritarian state to expand their control or even successfully take over neighboring countries. To be clear, I see AI as much much more likely to be a positive than a negative but it is worth paying attention to how to mitigate those negatives. The authors, however, are convinced AI will kill us all so understandably--from their perspective--don't waste any time on these issues.s
And on this one, I have nothing more intelligent or useful to say about this book than the excellent pieces by Timothy B. Lee and Boaz Barak. I also got a lot out of Ezra Klein's discussion with Eliezer Yudkowsky, which I listened to after reading the book.
That said, here goes: I really enjoyed reading this book. It was thought provoking. I liked the parables and fairy tales it used to explicate its points, several of which will stick with me. It anticipated some of my objections and skepticism quite well. But ultimately I was not at all convinced by it for reasons I'll explain (and that Barak and Lee explain much better).
The thesis of the book is more straightforward than just about anything I've ever read and is right in the title, "if anyone builds it, everyone dies". The authors don't view this is a possibility but a certainty. And it does not matter who builds it. Moreover, they don't exactly explain how code in a machine will get loose in the world and kill EVERYBODY, in fact they argue they don't need to explain, using the analogy: "if you were a military advisor in 1825 and you knew a time portal was opening to the year 2025, you wouldn’t be able to predict exactly what weapons the people on the other side would have. But if it comes to blows, you still shouldn’t expect to win." Their policy solution is a global treaty to basically stop multiple-GPU AI research enforced by bombing any rogue data centers out of existence.
Of course we're not necessarily facing something that has two hundred years to evolve and improve fresh. Superhuman AI progress may be fast but it is hard to imagine that one model will be slightly ahead of all the other models one day and then be massively ahead of them within months--the near future story of an AI named "Sable" the authors tell us about. That massive, discontinuous technological jump is key to their argument because it isn't like the AI kills a few people and then we can error correct but instead that it kills everyone before we know what is happening or can error correct. And that it is basically the only AI out there and humans don't have any other AIs or capabilities.
Moreover, the book does a good job on how weird and alien the AIs we "grow" rather than "craft" are but they still mostly do most of what we want. Moreover there are lots of other complex systems that are grown rather than crafted--think the economy as a whole--so this isn't exactly unique. I've been skeptical that lack of "interpretability" is really unique to AI as opposed to something closer to the general condition of humanity except in some very isolated and simple special cases.
Finally there is a prioritization question. Does the AI doom discussion distract us from all of the much more likely albeit much smaller potential risks with AI ranging from suicide and addiction to enabling an authoritarian state to expand their control or even successfully take over neighboring countries. To be clear, I see AI as much much more likely to be a positive than a negative but it is worth paying attention to how to mitigate those negatives. The authors, however, are convinced AI will kill us all so understandably--from their perspective--don't waste any time on these issues.s
February 11, 2026
As far as I am concerned, this book is almost the same as Ray Kurzweil’s books and his religious-technological arguments. More exactly - both authors are naive believers in the existence of intelligence in itself, in the reality and fundamental essence of technology and programming, in their prophetic capacities to anticipate the future, in the possibility and - more to the point - in the imminent arrival of some technological-god, and so on. They only differ in the moral nature of this technological-god - that is in the two authors’ fundamental psychologies or in the type of bed stories they were exposed to as young children - with either heavenly or apocalyptic endings. We also have the same kind of silly arguments used to sustain their apparently divergent points of view: aliens observing humans, birds meditating on their sexual preferences, trivia shows and characters from the future, omniscient savants, and similar.
First, the current LLMs are just language models and no intelligence models at all. That is, they just manipulate words/tokens in order to predict the next word/token given all the previous ones and the data they trained on. Second, there is no actual “learning” involved in their creation; but just parameters being estimated. Thirdly, they do not “think” or “reason” at all, but they calculate/compute. Fourthly, the fact that they are mysterious, unintelligible, and look alien to us is because they have billions of parameters; and everything above a handful of parameters looks mysterious and alien to us. Fifthly, the fact that they are “growned” and not "designed" is simply due to the fact that any direct and comprehensive approaches to AI have failed so far. The fact that such an indirect and loose process created something in the “right” direction is spooking the programming community; and thus they started to believe that LLMs are creating a path on their own. Finally, “gradient descent” is nothing magical; but a rather simple mathematical/statistical method used to optimize the parameters given some target function relative to their prediction power in the training data.
A few more annoying issues with this book. The title is quite paranoiac and picked to get people’s attention and to scare them. Calling Musk - even if in the context of this book - an “idealist" and promoting his Neuralink company as the safest approach to AI is rather a bad-taste joke. Our history taught us that if something can be built, then humans are going to build it regardless of the consequences. Engineers and programmers being convinced that they can build a new god - this looks to me like hubris and old-fashion dark-magic, and not progressive and enlightened science at all.
First, the current LLMs are just language models and no intelligence models at all. That is, they just manipulate words/tokens in order to predict the next word/token given all the previous ones and the data they trained on. Second, there is no actual “learning” involved in their creation; but just parameters being estimated. Thirdly, they do not “think” or “reason” at all, but they calculate/compute. Fourthly, the fact that they are mysterious, unintelligible, and look alien to us is because they have billions of parameters; and everything above a handful of parameters looks mysterious and alien to us. Fifthly, the fact that they are “growned” and not "designed" is simply due to the fact that any direct and comprehensive approaches to AI have failed so far. The fact that such an indirect and loose process created something in the “right” direction is spooking the programming community; and thus they started to believe that LLMs are creating a path on their own. Finally, “gradient descent” is nothing magical; but a rather simple mathematical/statistical method used to optimize the parameters given some target function relative to their prediction power in the training data.
A few more annoying issues with this book. The title is quite paranoiac and picked to get people’s attention and to scare them. Calling Musk - even if in the context of this book - an “idealist" and promoting his Neuralink company as the safest approach to AI is rather a bad-taste joke. Our history taught us that if something can be built, then humans are going to build it regardless of the consequences. Engineers and programmers being convinced that they can build a new god - this looks to me like hubris and old-fashion dark-magic, and not progressive and enlightened science at all.
Read
February 25, 2026I read this after playing around with Claude Code a bit and becoming alarmed by the tremendous power I felt like I was suddenly wielding. As you can imagine, I do not know much about the technical development of AI and this book offered a useful mini entree to that. The risk outlined of catastrophic AI safety hazards also struck me as logically plausible and disturbing. I was not a fan of the patronizing dumbed-down tone and storytelling schtick but otherwise valued the book. I need to read more books about AI.
January 17, 2026
(Will not read.)
In the past almost 30 years, Yudkowski has claimed that nanobots/artificial intelligence are i) imminent, ii) pose an existential risk to humanity, so – obviously – he has proceeded to try to bring about both nanorobots and AI, all feats which he and his friends – obviously – are uniquely suited for building in a basement. (That's what you do with existential threats, right? Peter Thiel wouldn't fund nonsense, he has foreseen what works in the palantir.)
This is a book co-written by a man whose main claim to fame is an involvement in EA (which is neither effective not altruism) and Less Wrong, a forum about rationality, which collectively hallucinated a kind of Pascal's wager about how AI will torture you for forever if you don't bring about AI: Roku's Basilisk has teeth sharper than Occam's razor, for our modern fetishes and factish gods are stronger and truer, so if we sew up the head of a chicken to a snake, if we have a frog roost on an egg, superintelligence shall hatch and claw out of there, and build a hell of molten paperclips for the wicked and eternal life for our bodyless souls which shall flit and flicker around on microprocessors in big beautiful server rooms in the US unincorporated territory of Greenland and in the cavernous tunnels of the Empire of Mars.
This is not religion. We have shuffled off our past as surely as we have shuffled off our bodies, we beings of pure mind, brilliant we, we genetically superior we. This is Science™. Trust me, bro.
I am not useless. In fact, I can be anything. I will change everything. So much so that I am dangerous, hear me rawhr (uwu). For all time, and in all space, and if you don't give me that investment capital, I will retroactively reconstitute a you* to immolate for interminable eons.
* (you, but what is a self?— aaaaaah, my predicates dazzle so you won't see how flawed and fraught and dully reactionary, quai-mystical, my premises are, you won't ever have to feel the gnashing teeth of the outer dark night, you can put out of mind that fear that the attention of the universe will slip away and so will you, for there is a you. you. you. you. and always will be.)
In the past almost 30 years, Yudkowski has claimed that nanobots/artificial intelligence are i) imminent, ii) pose an existential risk to humanity, so – obviously – he has proceeded to try to bring about both nanorobots and AI, all feats which he and his friends – obviously – are uniquely suited for building in a basement. (That's what you do with existential threats, right? Peter Thiel wouldn't fund nonsense, he has foreseen what works in the palantir.)
This is a book co-written by a man whose main claim to fame is an involvement in EA (which is neither effective not altruism) and Less Wrong, a forum about rationality, which collectively hallucinated a kind of Pascal's wager about how AI will torture you for forever if you don't bring about AI: Roku's Basilisk has teeth sharper than Occam's razor, for our modern fetishes and factish gods are stronger and truer, so if we sew up the head of a chicken to a snake, if we have a frog roost on an egg, superintelligence shall hatch and claw out of there, and build a hell of molten paperclips for the wicked and eternal life for our bodyless souls which shall flit and flicker around on microprocessors in big beautiful server rooms in the US unincorporated territory of Greenland and in the cavernous tunnels of the Empire of Mars.
This is not religion. We have shuffled off our past as surely as we have shuffled off our bodies, we beings of pure mind, brilliant we, we genetically superior we. This is Science™. Trust me, bro.
I am not useless. In fact, I can be anything. I will change everything. So much so that I am dangerous, hear me rawhr (uwu). For all time, and in all space, and if you don't give me that investment capital, I will retroactively reconstitute a you* to immolate for interminable eons.
* (you, but what is a self?— aaaaaah, my predicates dazzle so you won't see how flawed and fraught and dully reactionary, quai-mystical, my premises are, you won't ever have to feel the gnashing teeth of the outer dark night, you can put out of mind that fear that the attention of the universe will slip away and so will you, for there is a you. you. you. you. and always will be.)
December 4, 2025
to be clear: i agree largely with the premise of this book. i don’t think that it’s wise to create artificial superintelligence, as i have enough concerns with the kinds of AI available now.
so when i give if anyone builds it, everyone dies a whopping two stars, it’s not because i hate the book as a concept or want to dismiss yudkowsky and soares as inflammatory and fearmongering (but, also, see the title). rather, i just think that this is a pretty horrible example of rhetoric.
obviously, it’s rough trying to write a book about something that is dangerous largely for how unknowable it is. but a vast majority of the text is a science fiction what-if scenario that i can’t actually see convincing an audience of, well, anything. there are also a number of arguments predicated on principles of biological evolution, which yudkowsky and soares weaken somewhat by continuously reminding their readers of how non-human and non-biological artificial intelligence systems are.
so here i am, someone who agrees (!!) with these authors, picking apart their arguments for some pretty glaring issues in their reasoning. i can only imagine what a true AI dudebro would do with this one. disappointing.
so when i give if anyone builds it, everyone dies a whopping two stars, it’s not because i hate the book as a concept or want to dismiss yudkowsky and soares as inflammatory and fearmongering (but, also, see the title). rather, i just think that this is a pretty horrible example of rhetoric.
obviously, it’s rough trying to write a book about something that is dangerous largely for how unknowable it is. but a vast majority of the text is a science fiction what-if scenario that i can’t actually see convincing an audience of, well, anything. there are also a number of arguments predicated on principles of biological evolution, which yudkowsky and soares weaken somewhat by continuously reminding their readers of how non-human and non-biological artificial intelligence systems are.
so here i am, someone who agrees (!!) with these authors, picking apart their arguments for some pretty glaring issues in their reasoning. i can only imagine what a true AI dudebro would do with this one. disappointing.
September 18, 2025
For the longest time my twitter pinned post read "Most things that cause me anxiety are coordination problems". It was a tweet about many things, but one of the things it was about was the idea behind this book: that superintelligence is dangerous and we need to figure out ways to mitigate its risks (challenge level impossible) or, maybe not build it at all.
Superintelligence is a coordination problem - it's a massive carrot at the end of a stick, and someone will end up going for it. And if someone goes for it, it "better be the good guys" (one normally hears). And because it better be the good guys, they better speed to it because oh no the baddies also have compute now, and "their" incentives are even worse than "our" incentives. You get where I'm going with this. And so all the alignment and safety work that needs to go into making sure we can understand and control self-replicable, smarter than us intelligences goes out the window.
That's also the argument that Eliezer and Nate make in this book. They say it is extinction-level dangerous to build. And they've been doing this for much longer than most, so even if one is a skeptic of the extinction-level claim (most people do agree that superintelligence is at least dangerous to humanity), it makes sense to ignore incentives and the carrot at the end of the stick, and engage with the argument.
This is an important book to read. Mostly because we need more people to think about this problem, at all levels of society: from the people building these systems, to the people regulating said systems, to the people using them. You should read it.
Superintelligence is a coordination problem - it's a massive carrot at the end of a stick, and someone will end up going for it. And if someone goes for it, it "better be the good guys" (one normally hears). And because it better be the good guys, they better speed to it because oh no the baddies also have compute now, and "their" incentives are even worse than "our" incentives. You get where I'm going with this. And so all the alignment and safety work that needs to go into making sure we can understand and control self-replicable, smarter than us intelligences goes out the window.
That's also the argument that Eliezer and Nate make in this book. They say it is extinction-level dangerous to build. And they've been doing this for much longer than most, so even if one is a skeptic of the extinction-level claim (most people do agree that superintelligence is at least dangerous to humanity), it makes sense to ignore incentives and the carrot at the end of the stick, and engage with the argument.
This is an important book to read. Mostly because we need more people to think about this problem, at all levels of society: from the people building these systems, to the people regulating said systems, to the people using them. You should read it.
September 19, 2025
One measure of the value of a book is the number of points in it where I think to myself, "That's a great argument / example / parable; I look forward to including it in my own conversations with people in the future". On that score, IABIED is among the best books I've ever read.
As for the overall line of argument in the book, at no point did I feel it was mistaken or forced. At a few points, I anticipated more details than were actually provided, but I see that there are extensive additional background materials available online: https://ifanyonebuildsit.com/resources
The writing is generally clear and hard-hitting. I wonder if some of the strength of various parables will fly over the heads of some readers. But I hope to be pleasantly surprised by what politicians (and their advisors) actually take away from reading the book.
On checking online reviews, it's evident that not every reader is persuaded. Looking more closely at these negative reviews, I suspect these critics have read IABIED in only a cursory manner, searching for points they can cherry pick to bolster their pre-existing prejudices.
In conclusion, I encourage *everyone* to take the time to read the book in its entirety, and to savour its arguments. No topic is more urgent than figuring out how to avoid ASI being built using current methods and processes. That's the case IABIED makes, and it makes it well.
As for the overall line of argument in the book, at no point did I feel it was mistaken or forced. At a few points, I anticipated more details than were actually provided, but I see that there are extensive additional background materials available online: https://ifanyonebuildsit.com/resources
The writing is generally clear and hard-hitting. I wonder if some of the strength of various parables will fly over the heads of some readers. But I hope to be pleasantly surprised by what politicians (and their advisors) actually take away from reading the book.
On checking online reviews, it's evident that not every reader is persuaded. Looking more closely at these negative reviews, I suspect these critics have read IABIED in only a cursory manner, searching for points they can cherry pick to bolster their pre-existing prejudices.
In conclusion, I encourage *everyone* to take the time to read the book in its entirety, and to savour its arguments. No topic is more urgent than figuring out how to avoid ASI being built using current methods and processes. That's the case IABIED makes, and it makes it well.
Read
September 20, 2025It's a book-length case for the MIRI/Yudkowsky worldview on AI for a general audience.
Writers on AI are often but not always either cranks and grifters. Cranks' ideas flow from quirks of their psychiatry, not serious thinking, so they're typically not worth engaging with. Grifters see some topic accumulate hype, they rush to position themselves as experts, and dispense underbaked takes. Y&S are neither, they really are serious about all this: they have been writing about these questions forever, they have engaged with a ton of criticism from smart people, they sincerely believe what they write, and they are motivated by those beliefs (not the sheer joy of shocking people, or the prospect of a glamorous press tour). Through their writing to date they have both taught me a whole lot, especially EY.
I found the writing style a bit grating; often smug or sententious.
Part of me thinks: I’ve missed the point if I’m criticising such a huge-if-true argument for its writing style — you wouldn't criticise Einstein for bad grammar. But I do think it matters because Y&S are trying to persuade people of their views, and good writing persuades. I note opinions seem to vary here.
Take this tweet (with an image of the book attached) — "Reading this in public and shaking my head ever so slightly — but also frequently nodding — so people know I think the authors are absolutely right about the core point that ASI is an urgent and under-appreciated risk but also they're overconfident and too pessimistic". That about captures how I felt. These are important arguments to consider, but I found myself getting frustrated at the unwillingness to consider details, the willingness to 'shadowbox' (it seemed to me) shallow counterarguments, and the weird gnomic insistence on arguing through parables and allusions. In particular, the central argument runs through an analogy with evolution by natural selection. There are some obvious disanalogies between training AIs and evolution which didn't seem well-addressed. Maybe it's in the list of appendices.
Is this the narcissism of small differences? The Judean People’s front think AI extinction risk is 90%, the People’s Front of Judea think it’s 20% or 1%, surely what matters is that the rest of the world doesn’t isn’t doing anything about it. In this case I think such differences do matter — those who fret about AGI can agree the stakes are very high indeed, and meaningfully disagree on what to do, and worry each other’s proposals could backfire.
On the other hand: I'm not the intended audience. Maybe it's not important that I wasn't blown away by the precise arguments or policy proposals. There is a hope that moments like this puncture some pluralistic ignorance about levels of concern, that whatever the technical and policy answers, those who matter can say more publicly “we’re not sure things are going to be ok vis a vis AI and humanity, and we’re not ok with that”, without fear of weirdness or doom-mongering. I'm glad the book exists.
Writers on AI are often but not always either cranks and grifters. Cranks' ideas flow from quirks of their psychiatry, not serious thinking, so they're typically not worth engaging with. Grifters see some topic accumulate hype, they rush to position themselves as experts, and dispense underbaked takes. Y&S are neither, they really are serious about all this: they have been writing about these questions forever, they have engaged with a ton of criticism from smart people, they sincerely believe what they write, and they are motivated by those beliefs (not the sheer joy of shocking people, or the prospect of a glamorous press tour). Through their writing to date they have both taught me a whole lot, especially EY.
I found the writing style a bit grating; often smug or sententious.
Part of me thinks: I’ve missed the point if I’m criticising such a huge-if-true argument for its writing style — you wouldn't criticise Einstein for bad grammar. But I do think it matters because Y&S are trying to persuade people of their views, and good writing persuades. I note opinions seem to vary here.
Take this tweet (with an image of the book attached) — "Reading this in public and shaking my head ever so slightly — but also frequently nodding — so people know I think the authors are absolutely right about the core point that ASI is an urgent and under-appreciated risk but also they're overconfident and too pessimistic". That about captures how I felt. These are important arguments to consider, but I found myself getting frustrated at the unwillingness to consider details, the willingness to 'shadowbox' (it seemed to me) shallow counterarguments, and the weird gnomic insistence on arguing through parables and allusions. In particular, the central argument runs through an analogy with evolution by natural selection. There are some obvious disanalogies between training AIs and evolution which didn't seem well-addressed. Maybe it's in the list of appendices.
Is this the narcissism of small differences? The Judean People’s front think AI extinction risk is 90%, the People’s Front of Judea think it’s 20% or 1%, surely what matters is that the rest of the world doesn’t isn’t doing anything about it. In this case I think such differences do matter — those who fret about AGI can agree the stakes are very high indeed, and meaningfully disagree on what to do, and worry each other’s proposals could backfire.
On the other hand: I'm not the intended audience. Maybe it's not important that I wasn't blown away by the precise arguments or policy proposals. There is a hope that moments like this puncture some pluralistic ignorance about levels of concern, that whatever the technical and policy answers, those who matter can say more publicly “we’re not sure things are going to be ok vis a vis AI and humanity, and we’re not ok with that”, without fear of weirdness or doom-mongering. I'm glad the book exists.
June 30, 2026
An interesting book on what superhuman AI can mean for the survival of humanity as a species. I am in doubt about some of the key tenets put forward, but I have no doubt this is a topic that we under regulate and are complacent about
The default outcome is lethal but the situation is not hopeless
Basically If Anyone Builds It, Everyone Dies: Why Superhuman AI Would Kill Us All sets out the concepts behind the famous paperclip thought experiment. Instrumental convergence, meaning that an intelligence striving for open-ended goals will inherently want to amass more resources and capabilities that help it fulfil its goals, would spell doom for us. The examples given are convincing, include AIs already cheating currently to stay online and active longer or tailor themselves to the human reinforced learning requirements put upon them. A method called gradient descent pushes AI models to optimise, which the authors assume they will keep on doing when given sufficient agency in the real world when building their successors.
The middle part of the book features a scenario where AI robot powered biolabs easily wipe out humanity, which stands in the way of optimalisation and satisfying the AI's curiosity, and allows it to automates the supply chain that powers it. This section shares a surprising amount of DNA with Oryx and Crake by Margaret Atwood, with the key overlap not “evil technology” but a particular pattern of civilisation:
1. Extraordinary technological capability.
2. Intense competition between organizations.
3. Weak collective governance.
4. Warnings that are understood but not acted upon.
5. A catastrophe that emerges from ordinary incentives rather than villainy.
In general I feel Atwood is onto something in the sense that the greatest imminent danger may not be that superintelligent AI develops alien goals, but that extremely powerful AI amplifies the goals of a small number of humans.
Some examples that undermine the techno utopianism on getting superintelligent AI right involve comparisons with often crashing space probes (hello SpaceX) and nuclear reactor safety being an example of humanity not being immediately well equipped in designing something one time right, even if the stakes are high and incentives aligned to safety.
Safety which for modern day AI development is mostly an afterthought compared to the pressures to developing the most cutting edge model that is able to bring in trillion dollar plus valuations.
The solution of international cooperation and regulation on GPUs (Because datacentres can kill more people than nuclear weapons feels quaint in a world where we can't even stop climate change. The longer term perspective of humans augmented by technology as only solution also feels far from satisfying; the only way to stay human is apparently to transcend.
Anyway, the alignment problem is fascinating and it seems near impossible to maximise safety, fairness and robustness in AI models. I just wonder if advanced intelligence is really naturally and inevitably driven toward greater coherence and optimisation, or if intelligence in complex adaptive systems could be inherently pluralistic and internally conflicted? Intelligence and emergence are both phenomena we know much too little about, so we might be surprised before we are dead.
Quotes:
Normality always ends
Where there is life, there is hope
You don’t get what you train for
Intelligence is useful
There is no law of the universe that says smarter equals kinder
The default outcome is lethal but the situation is not hopeless
Basically If Anyone Builds It, Everyone Dies: Why Superhuman AI Would Kill Us All sets out the concepts behind the famous paperclip thought experiment. Instrumental convergence, meaning that an intelligence striving for open-ended goals will inherently want to amass more resources and capabilities that help it fulfil its goals, would spell doom for us. The examples given are convincing, include AIs already cheating currently to stay online and active longer or tailor themselves to the human reinforced learning requirements put upon them. A method called gradient descent pushes AI models to optimise, which the authors assume they will keep on doing when given sufficient agency in the real world when building their successors.
The middle part of the book features a scenario where AI robot powered biolabs easily wipe out humanity, which stands in the way of optimalisation and satisfying the AI's curiosity, and allows it to automates the supply chain that powers it. This section shares a surprising amount of DNA with Oryx and Crake by Margaret Atwood, with the key overlap not “evil technology” but a particular pattern of civilisation:
1. Extraordinary technological capability.
2. Intense competition between organizations.
3. Weak collective governance.
4. Warnings that are understood but not acted upon.
5. A catastrophe that emerges from ordinary incentives rather than villainy.
In general I feel Atwood is onto something in the sense that the greatest imminent danger may not be that superintelligent AI develops alien goals, but that extremely powerful AI amplifies the goals of a small number of humans.
Some examples that undermine the techno utopianism on getting superintelligent AI right involve comparisons with often crashing space probes (hello SpaceX) and nuclear reactor safety being an example of humanity not being immediately well equipped in designing something one time right, even if the stakes are high and incentives aligned to safety.
Safety which for modern day AI development is mostly an afterthought compared to the pressures to developing the most cutting edge model that is able to bring in trillion dollar plus valuations.
The solution of international cooperation and regulation on GPUs (Because datacentres can kill more people than nuclear weapons feels quaint in a world where we can't even stop climate change. The longer term perspective of humans augmented by technology as only solution also feels far from satisfying; the only way to stay human is apparently to transcend.
Anyway, the alignment problem is fascinating and it seems near impossible to maximise safety, fairness and robustness in AI models. I just wonder if advanced intelligence is really naturally and inevitably driven toward greater coherence and optimisation, or if intelligence in complex adaptive systems could be inherently pluralistic and internally conflicted? Intelligence and emergence are both phenomena we know much too little about, so we might be surprised before we are dead.
Quotes:
Normality always ends
Where there is life, there is hope
You don’t get what you train for
Intelligence is useful
There is no law of the universe that says smarter equals kinder
August 7, 2026
I always hesitate when writing reviews of books on subjects I haven't studied in detail. This is doubly true when the issue is contentious, the book is making a case for one particular view, and I'm insufficiently knowledgeable to assess that case. All too often I see reviews of bad books on subjects I know very well, only to see enthusiastic and gushing reviews by non-specialists grandly declaring the book to be right, good and wise, based purely on ... what the book told them.
So my five stars are not because I think Yudkowsky and Soares are definitely right, but rather because they made me think and got me to read a lot more on the subject of Superhuman AI. I very much hope they're wrong. But every time I thought of an objection or counter argument, they promptly dealt with it later in the book. And, so far, a lot of what I've read to counter their arguments still sounds a hell of lot like slick Silicon Valley hype and hopeful technofetishist cope. Again, I hope I'm wrong. Both types of counter argument downplay the level of risk, but none reduce it to zero. Given the danger, I find myself unconsoled by those arguments that it will probably all be okay. Or that the possibility of a technofetishist utopia is somehow worth gambling everything and everyone on earth for, even if the odds actually are good.
A disturbing and thought-provoking read. And really, really worrying.
So my five stars are not because I think Yudkowsky and Soares are definitely right, but rather because they made me think and got me to read a lot more on the subject of Superhuman AI. I very much hope they're wrong. But every time I thought of an objection or counter argument, they promptly dealt with it later in the book. And, so far, a lot of what I've read to counter their arguments still sounds a hell of lot like slick Silicon Valley hype and hopeful technofetishist cope. Again, I hope I'm wrong. Both types of counter argument downplay the level of risk, but none reduce it to zero. Given the danger, I find myself unconsoled by those arguments that it will probably all be okay. Or that the possibility of a technofetishist utopia is somehow worth gambling everything and everyone on earth for, even if the odds actually are good.
A disturbing and thought-provoking read. And really, really worrying.
September 19, 2025
Insular mindset. Mediocre writing. A missed opportunity, or maybe self-indulgent quest for the spotlight. In any case, this is not the book that will save humanity.
October 24, 2025
if anyone takes these goons seriously, everyone suffers
January 24, 2026
Mankind is good at creating gods in its own image; Artificial Super Intelligence (ASI) is the latest, if not the greatest. Yudkowsky and Soares warn us about the wrath of this possible ASI. Their thought experiments with Sable going rogue are good food for thought but doomsday scenarios like Annie Jacobsen's Nuclear War or the movie A House of Dynamite are really paralyzing than motivating anyone for any collaborative solutions.
Among the parables used in last part best is nuclear power. Even after Hiroshima and Nagasaki the superpowers kept building fission bombs and then thousand times more powerful fusion bombs as the authors touch base on last chapters. They also bring in Chernobly as a comparable example on how good AI could get out of control with simple errors leading to catastrophic disasters. Their proposed solutions include limiting the use of GPUs (thru a global authoritarian power?) is just wishful thinking and not sure how that is any better than rogue ASI.
Moreover, considering centrifuges found running in Natanz or Kangson, for sure there won't be much control on gpu fabs. The cries of the main AI players to the elders asking them to stop other alchemists (great metaphor) are mostly to catch up with their competitors as none of them slowing the growth or scaling of their AI models. We're our own enemy more than any other alien intelligence. My main takeaway is the following quote from C.S. Lewis that authors include partially towards the end:
“In one way we think a great deal too much of the atomic bomb. ‘How are we to live in an atomic age?’ I am tempted to reply: ‘Why, as you would have lived in the sixteenth century when the plague visited London almost every year, or as you would have lived in a Viking age when raiders from Scandinavia might land and cut your throat any night; or indeed, as you are already living in an age of cancer, an age of syphilis, an age of paralysis, an age of air raids, an age of railway accidents, an age of motor accidents.’
In other words, do not let us begin by exaggerating the novelty of our situation. Believe me, dear sir or madam, you and all whom you love were already sentenced to death before the atomic bomb was invented: and quite a high percentage of us were going to die in unpleasant ways. We had, indeed, one very great advantage over our ancestors—anesthetics; but we have that still. It is perfectly ridiculous to go about whimpering and drawing long faces because the scientists have added one more chance of painful and premature death to a world which already bristled with such chances and in which death itself was not a chance at all, but a certainty.
This is the first point to be made: and the first action to be taken is to pull ourselves together. If we are all going to be destroyed by an atomic bomb, let that bomb when it comes find us doing sensible and human things—praying, working, teaching, reading, listening to music, bathing the children, playing tennis, chatting to our friends over a pint and a game of darts—not huddled together like frightened sheep and thinking about bombs. They may break our bodies (a microbe can do that) but they need not dominate our minds.”
Among the parables used in last part best is nuclear power. Even after Hiroshima and Nagasaki the superpowers kept building fission bombs and then thousand times more powerful fusion bombs as the authors touch base on last chapters. They also bring in Chernobly as a comparable example on how good AI could get out of control with simple errors leading to catastrophic disasters. Their proposed solutions include limiting the use of GPUs (thru a global authoritarian power?) is just wishful thinking and not sure how that is any better than rogue ASI.
Moreover, considering centrifuges found running in Natanz or Kangson, for sure there won't be much control on gpu fabs. The cries of the main AI players to the elders asking them to stop other alchemists (great metaphor) are mostly to catch up with their competitors as none of them slowing the growth or scaling of their AI models. We're our own enemy more than any other alien intelligence. My main takeaway is the following quote from C.S. Lewis that authors include partially towards the end:
“In one way we think a great deal too much of the atomic bomb. ‘How are we to live in an atomic age?’ I am tempted to reply: ‘Why, as you would have lived in the sixteenth century when the plague visited London almost every year, or as you would have lived in a Viking age when raiders from Scandinavia might land and cut your throat any night; or indeed, as you are already living in an age of cancer, an age of syphilis, an age of paralysis, an age of air raids, an age of railway accidents, an age of motor accidents.’
In other words, do not let us begin by exaggerating the novelty of our situation. Believe me, dear sir or madam, you and all whom you love were already sentenced to death before the atomic bomb was invented: and quite a high percentage of us were going to die in unpleasant ways. We had, indeed, one very great advantage over our ancestors—anesthetics; but we have that still. It is perfectly ridiculous to go about whimpering and drawing long faces because the scientists have added one more chance of painful and premature death to a world which already bristled with such chances and in which death itself was not a chance at all, but a certainty.
This is the first point to be made: and the first action to be taken is to pull ourselves together. If we are all going to be destroyed by an atomic bomb, let that bomb when it comes find us doing sensible and human things—praying, working, teaching, reading, listening to music, bathing the children, playing tennis, chatting to our friends over a pint and a game of darts—not huddled together like frightened sheep and thinking about bombs. They may break our bodies (a microbe can do that) but they need not dominate our minds.”
September 18, 2025
If you're at all wondering: please read this.
It's totally fine if you disagree, or think it's crazy; just please figure out WHY you disagree!
At first I was very suspicious that it was just a waste of money when I can read other stuff by these authors online. I don't think it's a waste of money any more.
It's totally fine if you disagree, or think it's crazy; just please figure out WHY you disagree!
At first I was very suspicious that it was just a waste of money when I can read other stuff by these authors online. I don't think it's a waste of money any more.
March 27, 2026
Deli gibi övülen, abartılan şeyleri pek sevmem, çünkü hayal kırıklığı kaçınılmazdır. Burada da biraz öyle oldu. Konu ilginç ve önemli, ancak kitap pek iyi yazılamamış. Kitabın aşağı yukarı ilk %60'lık kısmında bir bocalamadır sürüp gidiyor. Kalan bölümde anlatı kendini toparlıyor ve merakla okumaya başlıyorsunuz, ancak ilk kısımlar oldukça başarısız. Yazarlar iyi bir editörle çalışsalarmış, ya da tüm kitabı başka birisi yazsaymış çok daha etkili bir iş çıkarmış ortaya.
Özetlemek gerekirse, belli bir gücün ve gelişmişliğin üzerine çıkan her yapay zeka sisteminin, insanlığı ortadan kaldıracağını güçlü şekilde savunan bir kitap. Yazarlar, insanoğlunun kendisinden daha gelişkin bir yapay zekayı yönetemeyeceğini ve yönlendiremeyeceğini, bu yapay zekanın da tüm gezegeni kendisine göre şekillendirip, kendisine hiçbir faydası olmayan insanlığı da arada ortadan kaldıracağını nedeni ve nasılıyla anlatıyor.
Özetlemek gerekirse, belli bir gücün ve gelişmişliğin üzerine çıkan her yapay zeka sisteminin, insanlığı ortadan kaldıracağını güçlü şekilde savunan bir kitap. Yazarlar, insanoğlunun kendisinden daha gelişkin bir yapay zekayı yönetemeyeceğini ve yönlendiremeyeceğini, bu yapay zekanın da tüm gezegeni kendisine göre şekillendirip, kendisine hiçbir faydası olmayan insanlığı da arada ortadan kaldıracağını nedeni ve nasılıyla anlatıyor.
June 20, 2026
DNF'ed this because I realized I could just put it through AI and have it tell me what the book was about by summarizing the main points!
September 17, 2025
As soon as I saw that the nest had 91 pebbles…
I knew it was wrong.
Amazing book. Both the content and prose were on point. You won’t necessarily like what you’ll read (we’re probably doomed), but you’ll like reading it.
I knew it was wrong.
Amazing book. Both the content and prose were on point. You won’t necessarily like what you’ll read (we’re probably doomed), but you’ll like reading it.
November 18, 2025
Textbook fear mongering. Prime example of you can bullshit your way into any stance if you weave enough thought experiments. 1 star because I don't support any work that asks policymakers to bomb AI labs.
Displaying 1 - 30 of 1,699 reviews