What do you think?
Rate this book
Situational Awareness: The Decade Ahead
165 pages, ebook
Published June 1, 2024
32 people are currently reading
493 people want to read
Ratings & Reviews
Friends & Following
Create a free account to discover what your friends think of this book!
Community Reviews
Displaying 1 - 30 of 40 reviews
January 23, 2025
I saw a reference to this 165-page paper in a Guardian article earlier today. I downloaded it (here) and just couldn't stop reading until I'd finished.
If you're working with modern AI, the paper is chock-full of useful information. Leopold Aschenbrenner, a wunderkind who allegedly was fired from OpenAI a few weeks ago for leaking secrets, is part of the inner circle of AI research and loves to shoot his mouth off. He frequently repeats himself (the paper does not appear to have received much editing), but I didn't mind: it's a quick and easy read. He gives an excellent up-to-the-minute snapshot of where we are with AI, with many useful references. To name just a few things that particularly piqued my interest, I need to follow up on Chain of Thought reasoning, which I see we're underutilising, and majority voting, which we aren't using at all. There's a fascinating summary of work reported in the Gemini 1.5 paper, which I was embarrassed not to have read, in particular mentioning the remarkable results they have achieved in underresourced languages. I have already passed on the reference to colleagues working in that area.
I had not appreciated either how quickly things are moving towards construction of very large AI clusters costing on the order of hundreds of billions or even trillions of dollars. These will consume monstrous amounts of electrical power, amounts comparable with the requirements of small states, and apparently there is a frantic bidding war going on to lock down power supply contracts. It is all happening right now. The author looks at the technical advances and the infrastructure development program, and puts together a plausible argument for AGI before the end of the decade, most likely 2027-28, and superintelligence a year or two later. It may not happen. But the extrapolations no longer seem at all far-fetched, and gigantic sums of money are being bet on the assumption that they are correct. Everything he said meshed well with my own experiences of using AI. I will be surprised if it does not happen.
In the last sections, however, I must unfortunately say that I started rolling my eyes. He tells us that this technology is of inestimable commercial and military importance and that the US currently has a substantial lead, but that the key data could very easily be stolen, in particular by the Chinese. So far, I agree. But then he says the only solution is for a new Manhattan project with a similar level of security, so that the US can keep control of these vital secrets.
Sigh. Not a word about the rather high likelihood that the next President of the United States could be the agent of an unfriendly power and entirely ready to pass on those secrets for his own personal gain. Honestly, how naïve can you get? But whatever you may think of Aschenbrenner's abilities as a prose stylist or a political analyst, it's a must-read. Check it out.
____________
And a couple of days later...
This paper is the most disquieting thing I've seen for a while. One wonders what the author's real purpose was in releasing it. He was allegedly fired from OpenAI for leaking secrets, and now he's trying to persuade the US government to put his former employers in an ultra-secure facility in the desert to stop them from leaking secrets. Would it be too much of a stretch to wonder if this could be an attempted act of revenge? Aschenbrenner stresses that everything in the paper comes from publicly available sources, which in a narrow sense is probably true, but many dots are being connected here.
What everyone in Silicon Valley is discussing, we hear, is how to build the trillion-dollar AI hubs that are going to be needed to reach superintelligence. It has to be done very quickly, given that there's a frantic race going on and people believe the goal can be reached in four or five years. The most difficult problem is finding enough energy to power them; we're talking about an appreciable fraction of the whole US consumption of electricity. There are many references, mostly indirect ones, to the ongoing negotiations with the Gulf states, which have large amounts of energy and money. It sounds like one scenario on the table is to build the trillion-dollar hubs there. Aschenbrenner, who constantly offers the parallel between AI in the 2020s and nuclear weapons in the 1940s and 50s, says this is equivalent to the US producing and storing its nukes in a foreign dictatorship it at best has marginal control over. It is not a ridiculous analogy. As he says, superintelligence is likely to make all existing weapon systems obsolete.
He doesn't quite say it in so many words, but he appears to like the idea of a second Trump term. Trump has no interest at all in protecting the environment, and he'd be happy with increasing US energy production, and CO2 emissions, by the 25-50% necessary to run these datacentres inside the US. Given that Trump is obviously under the control of Vladimir Putin, Russia would get the technology at the same time. But when you're in a sufficiently realpolitik frame of mind, this isn't necessarily a bad thing. If the US did what Aschenbrenner ostensibly wants it to do, and somehow managed to lock down these incredibly valuable secrets so that no one could get at them, we'd have a highly unstable situation. Russia would be sitting on a nuclear arsenal they knew would soon become obsolete, and that could provoke all kinds of desperate measures. It's probably safer for everyone to get the technology at the same time and maintain a level playing field. Needless to say, this is a least bad scenario: a world where climate change has been drastically accelerated so that the US, Russia and presumably China, all under authoritarian governments, will have absolute technological dominance. But it's better than all-out nuclear war.
What interesting times we're living in.
____________
And a week after that...
I discussed Aschenbrenner's essay this afternoon with a friend who knows a lot more than I do about political and defence issues. He doesn't think the material about China in the later sections makes much sense, and wonders if its real purpose might not be to get Trump's attention.
The Audience of One, as people are starting to call it...
____________
From this Guardian article:
Say what you will about Aschenbrenner, his recommendations regarding AI architectures do indeed turn out to be pretty good. Take a look at our new paper.
____________
From today's Guardian article :
If you're working with modern AI, the paper is chock-full of useful information. Leopold Aschenbrenner, a wunderkind who allegedly was fired from OpenAI a few weeks ago for leaking secrets, is part of the inner circle of AI research and loves to shoot his mouth off. He frequently repeats himself (the paper does not appear to have received much editing), but I didn't mind: it's a quick and easy read. He gives an excellent up-to-the-minute snapshot of where we are with AI, with many useful references. To name just a few things that particularly piqued my interest, I need to follow up on Chain of Thought reasoning, which I see we're underutilising, and majority voting, which we aren't using at all. There's a fascinating summary of work reported in the Gemini 1.5 paper, which I was embarrassed not to have read, in particular mentioning the remarkable results they have achieved in underresourced languages. I have already passed on the reference to colleagues working in that area.
I had not appreciated either how quickly things are moving towards construction of very large AI clusters costing on the order of hundreds of billions or even trillions of dollars. These will consume monstrous amounts of electrical power, amounts comparable with the requirements of small states, and apparently there is a frantic bidding war going on to lock down power supply contracts. It is all happening right now. The author looks at the technical advances and the infrastructure development program, and puts together a plausible argument for AGI before the end of the decade, most likely 2027-28, and superintelligence a year or two later. It may not happen. But the extrapolations no longer seem at all far-fetched, and gigantic sums of money are being bet on the assumption that they are correct. Everything he said meshed well with my own experiences of using AI. I will be surprised if it does not happen.
In the last sections, however, I must unfortunately say that I started rolling my eyes. He tells us that this technology is of inestimable commercial and military importance and that the US currently has a substantial lead, but that the key data could very easily be stolen, in particular by the Chinese. So far, I agree. But then he says the only solution is for a new Manhattan project with a similar level of security, so that the US can keep control of these vital secrets.
Sigh. Not a word about the rather high likelihood that the next President of the United States could be the agent of an unfriendly power and entirely ready to pass on those secrets for his own personal gain. Honestly, how naïve can you get? But whatever you may think of Aschenbrenner's abilities as a prose stylist or a political analyst, it's a must-read. Check it out.
____________
And a couple of days later...
This paper is the most disquieting thing I've seen for a while. One wonders what the author's real purpose was in releasing it. He was allegedly fired from OpenAI for leaking secrets, and now he's trying to persuade the US government to put his former employers in an ultra-secure facility in the desert to stop them from leaking secrets. Would it be too much of a stretch to wonder if this could be an attempted act of revenge? Aschenbrenner stresses that everything in the paper comes from publicly available sources, which in a narrow sense is probably true, but many dots are being connected here.
What everyone in Silicon Valley is discussing, we hear, is how to build the trillion-dollar AI hubs that are going to be needed to reach superintelligence. It has to be done very quickly, given that there's a frantic race going on and people believe the goal can be reached in four or five years. The most difficult problem is finding enough energy to power them; we're talking about an appreciable fraction of the whole US consumption of electricity. There are many references, mostly indirect ones, to the ongoing negotiations with the Gulf states, which have large amounts of energy and money. It sounds like one scenario on the table is to build the trillion-dollar hubs there. Aschenbrenner, who constantly offers the parallel between AI in the 2020s and nuclear weapons in the 1940s and 50s, says this is equivalent to the US producing and storing its nukes in a foreign dictatorship it at best has marginal control over. It is not a ridiculous analogy. As he says, superintelligence is likely to make all existing weapon systems obsolete.
He doesn't quite say it in so many words, but he appears to like the idea of a second Trump term. Trump has no interest at all in protecting the environment, and he'd be happy with increasing US energy production, and CO2 emissions, by the 25-50% necessary to run these datacentres inside the US. Given that Trump is obviously under the control of Vladimir Putin, Russia would get the technology at the same time. But when you're in a sufficiently realpolitik frame of mind, this isn't necessarily a bad thing. If the US did what Aschenbrenner ostensibly wants it to do, and somehow managed to lock down these incredibly valuable secrets so that no one could get at them, we'd have a highly unstable situation. Russia would be sitting on a nuclear arsenal they knew would soon become obsolete, and that could provoke all kinds of desperate measures. It's probably safer for everyone to get the technology at the same time and maintain a level playing field. Needless to say, this is a least bad scenario: a world where climate change has been drastically accelerated so that the US, Russia and presumably China, all under authoritarian governments, will have absolute technological dominance. But it's better than all-out nuclear war.
What interesting times we're living in.
____________
And a week after that...
I discussed Aschenbrenner's essay this afternoon with a friend who knows a lot more than I do about political and defence issues. He doesn't think the material about China in the later sections makes much sense, and wonders if its real purpose might not be to get Trump's attention.
The Audience of One, as people are starting to call it...
____________
From this Guardian article:
In Trump’s America, there will be hard choices for everyone, including the billionaires. Though it may be less hard for them. Vance has said he wants to deregulate crypto and unshackle AI. He’s said he’d dismantle Biden’s attempts to place safeguards around AI development.____________
Say what you will about Aschenbrenner, his recommendations regarding AI architectures do indeed turn out to be pretty good. Take a look at our new paper.
____________
From today's Guardian article :
He said the US needs to double its energy production, partly to fuel artificial intelligence. Trump said he will fast-track the approvals for new power plants, which companies can site next to their plants – something not currently possible under regulations. Worryingly, he declared that companies will be able to fuel it with anything they want, and have coal as a backup, “good, clean coal”.So far, Aschenbrenner's predictions seem remarkably accurate.
June 22, 2024
I disagree with the author on several things in the essay, but he says things that need to be said and propels important discussion points into the light I appreciate.
For a counter view, here are some criticisms pushing back on the claims made in the essay and popular views currently in SF. I like Sabine’s comment “I can’t see no end! Says man who earns money from seeing no end.” 😆 https://www.youtube.com/watch?v=xm1B3...
The Guardian review: "How’s this for a bombshell – the US must make AI its next Manhattan Project";
https://www.theguardian.com/commentis...
Youtube video of the author;
https://www.youtube.com/watch?v=zdbVt...
Link to the essay;
https://situational-awareness.ai/
Essay as PDF
https://situational-awareness.ai/wp-c...
For a counter view, here are some criticisms pushing back on the claims made in the essay and popular views currently in SF. I like Sabine’s comment “I can’t see no end! Says man who earns money from seeing no end.” 😆 https://www.youtube.com/watch?v=xm1B3...
The Guardian review: "How’s this for a bombshell – the US must make AI its next Manhattan Project";
https://www.theguardian.com/commentis...
Youtube video of the author;
https://www.youtube.com/watch?v=zdbVt...
Link to the essay;
https://situational-awareness.ai/
Essay as PDF
https://situational-awareness.ai/wp-c...
June 19, 2024
Aschenbrenner is telling the inside-story of AI (he had a job at OpenAI). What are we to expect in the next decade? Not surprisingly he said that we will get AGI and then superintelligence. It will cost a hell lot of energy.
Security is bad and we better watch out. If the Chinese steal our (meaning theirs - the SF-Americans) technology the end of the world will come. That might come anyway. But in the hands of Americans the world is more likely to survive. If I understood him correctly.
Lots of parallels to the development of the Atom bomb. (What if they had not managed to keep Fermi silent?)
It is not a bad book. He even has a Goethe quote. But really nothing you did not know before.
Security is bad and we better watch out. If the Chinese steal our (meaning theirs - the SF-Americans) technology the end of the world will come. That might come anyway. But in the hands of Americans the world is more likely to survive. If I understood him correctly.
Lots of parallels to the development of the Atom bomb. (What if they had not managed to keep Fermi silent?)
It is not a bad book. He even has a Goethe quote. But really nothing you did not know before.
July 10, 2024
So yes, this is quite readable and I appreciated some nuggets of information and some nice paper references. I didn’t know for example that Scott Aaronson had attached himself to OpenAI for modelling AI alignment. It is overall a wildly non-objective paper (more an essay) and it is so US-centric and tries so hard to sound über-patriotic that it not only ignores the rest of the world but really makes it sound like the US government is the white knight in shiny armour that will sweep in and save us all. Sure.
What annoys me the most about most AI predictions is that they are always so reductive. The human as a factor is mostly left out of these predictive models (with exception of being treated as a job-holding/losing unit), things are optimised and accelerated for their own sake, but human satisfaction or even happiness or even life’s meaning is never a factor. These predictions are also not happening in a vacuum, but rather are being used as fuel for decision-making that is already happening behind closed doors in the centres of power: plans are being drawn up and being hastily implemented or at least pursued by money-(grubbing)/raising disruptors. The hard question of whether we should build things just because we can is never asked. AI is already being eaten by big tech, and "hype" papers such as this one reinforce a narrative that is arguably quite noxious to the already existing planet we live in. The thing is unless we live in a simulation and get to push a restart button any time soon, things are not looking out so fine, when you account for the resource-hunger of current AI technology.
As a quick note, I am definitely no technophobic Luddite. On the contrary, I have a PhD in computer science myself and have been both fascinated and intrigued by the concept (and implementation) of AI, since I was a little kid and started playing around with computers. I am also no doomer or a Yudkowsky fanboy. However, I am not really on board with this dystopian bright future we seem to be constructing for ourselves, where enough is never enough. Optimisation seems to be the end goal and we will never stop until humans are made irrelevant or at least easily discarded and until the planet is completely drained of its natural resources. It doesn't take a genius to see that this approach is unsustainable. We live in a real world with finite resources, where we are already facing enormous challenges posed by resource scarcity and climate change. Think about the many minerals needed for building our current hardware and batteries, such as lithium (transported by ships all the way from Chile to China) and cobalt (mined in the Congo at the cost of child labor and slave-like work).
It’s like watching kids racing each other to devour a cake and claiming they'll they save some for later whilst already licking the final crumbs off the plate. This arms-race mentality creates a sort of tunnel vision where greed and a sense of urgency eggs AI labs onwards without time for reflexion and consideration of what and how and why they are doing what they are doing. Aschenbrenner mentions different sorts of dangers involved in developing such a superintelligence, but still puts American national security above all else:
This idea of an optimising machine that eats itself and that keeps regurgitating an ever-improving version that will end up becoming the ultimate superintelligence is not only terrifying due to safety concerns (factor in human lives that are not only going to be economically affected by also physically endangered by such an intelligent form, regardless of whether it can be perfectly aligned to the interests of whoever is training or fine-tuning it) but also because if allowed to act as a free agent it will most probably use natural resources in very different ways than we do and not be as interested in consuming them in a human-sustainable manner but rather in a AI-sustainable one. Think about this Time’s article about a Bitcoin mine in Texas and the absurd levels of noise it currently generates (~85-90 dB) and how it is deeply affecting the community in which it was built (from causing heart issues, panic attacks, hearing loss, etc.). Imagine the scale of the issues that are going to materialise only from running these mega-clusters Aschenbrenner (and surely a lot of of other AI-fanboys) are wet-dreaming about.
So my take is that if we keep ignoring climate change as an extraneous variable and keep upon our current path, our future is not going to be one where we are schlepped around by flying surfboards (see Wall-E) and be entertained by optimised content (by multiple OOMs!) that keeps us "engaged" (complacent) within our bubbles, but one where will be back to sticks and stones and lucky if there are humans hanging around at all. And not necessarily because AI became so intelligent that it wiped us off the face of the earth, but rather because we ate the whole cake, while thinking there’d still be some for tomorrow.
What annoys me the most about most AI predictions is that they are always so reductive. The human as a factor is mostly left out of these predictive models (with exception of being treated as a job-holding/losing unit), things are optimised and accelerated for their own sake, but human satisfaction or even happiness or even life’s meaning is never a factor. These predictions are also not happening in a vacuum, but rather are being used as fuel for decision-making that is already happening behind closed doors in the centres of power: plans are being drawn up and being hastily implemented or at least pursued by money-(grubbing)/raising disruptors. The hard question of whether we should build things just because we can is never asked. AI is already being eaten by big tech, and "hype" papers such as this one reinforce a narrative that is arguably quite noxious to the already existing planet we live in. The thing is unless we live in a simulation and get to push a restart button any time soon, things are not looking out so fine, when you account for the resource-hunger of current AI technology.
As a quick note, I am definitely no technophobic Luddite. On the contrary, I have a PhD in computer science myself and have been both fascinated and intrigued by the concept (and implementation) of AI, since I was a little kid and started playing around with computers. I am also no doomer or a Yudkowsky fanboy. However, I am not really on board with this dystopian bright future we seem to be constructing for ourselves, where enough is never enough. Optimisation seems to be the end goal and we will never stop until humans are made irrelevant or at least easily discarded and until the planet is completely drained of its natural resources. It doesn't take a genius to see that this approach is unsustainable. We live in a real world with finite resources, where we are already facing enormous challenges posed by resource scarcity and climate change. Think about the many minerals needed for building our current hardware and batteries, such as lithium (transported by ships all the way from Chile to China) and cobalt (mined in the Congo at the cost of child labor and slave-like work).
It’s like watching kids racing each other to devour a cake and claiming they'll they save some for later whilst already licking the final crumbs off the plate. This arms-race mentality creates a sort of tunnel vision where greed and a sense of urgency eggs AI labs onwards without time for reflexion and consideration of what and how and why they are doing what they are doing. Aschenbrenner mentions different sorts of dangers involved in developing such a superintelligence, but still puts American national security above all else:
"American national security must come first, before the allure of free-flowing Middle Eastern cash, arcane regulation, or even, yes, admirable climate commitments"
This idea of an optimising machine that eats itself and that keeps regurgitating an ever-improving version that will end up becoming the ultimate superintelligence is not only terrifying due to safety concerns (factor in human lives that are not only going to be economically affected by also physically endangered by such an intelligent form, regardless of whether it can be perfectly aligned to the interests of whoever is training or fine-tuning it) but also because if allowed to act as a free agent it will most probably use natural resources in very different ways than we do and not be as interested in consuming them in a human-sustainable manner but rather in a AI-sustainable one. Think about this Time’s article about a Bitcoin mine in Texas and the absurd levels of noise it currently generates (~85-90 dB) and how it is deeply affecting the community in which it was built (from causing heart issues, panic attacks, hearing loss, etc.). Imagine the scale of the issues that are going to materialise only from running these mega-clusters Aschenbrenner (and surely a lot of of other AI-fanboys) are wet-dreaming about.
So my take is that if we keep ignoring climate change as an extraneous variable and keep upon our current path, our future is not going to be one where we are schlepped around by flying surfboards (see Wall-E) and be entertained by optimised content (by multiple OOMs!) that keeps us "engaged" (complacent) within our bubbles, but one where will be back to sticks and stones and lucky if there are humans hanging around at all. And not necessarily because AI became so intelligent that it wiped us off the face of the earth, but rather because we ate the whole cake, while thinking there’d still be some for tomorrow.
Read
December 31, 2024It is rare for me to withhold a rating but this time I have to.
July 29, 2024
3.5 stars. This lengthy but informative work stresses the future growth of AI.
From the technicalities of growth and bottlenecks to power consumption, safety, and security.
Soon, research will be increasingly automated. Which means computers can work around the clock to make their own leaps of progress, all faster than most of us regular people think possible. And what are we going to do about it?
From the technicalities of growth and bottlenecks to power consumption, safety, and security.
Soon, research will be increasingly automated. Which means computers can work around the clock to make their own leaps of progress, all faster than most of us regular people think possible. And what are we going to do about it?
July 12, 2024
Boring for 90% of the book, but the first 10 pages are worth it for how clearly and well thought the ideas are presented.
August 24, 2024
Trata de como o desenvolvimento de AI/AGI é comparável apenas - e ainda de longe - com o desenvolvimento de armas nucleares. E, como tal, o que pode ser feito para assegurar a direção com menor probabilidade de ~apocalipse.
O autor (extremamente qualificado) apresenta muitos fatos para apoiar suas afirmações, mas o ponto que tiro é que, mesmo que ele esteja muito otimista na velocidade do progresso em IA, se assumirmos qualquer taxa de progresso, um dia teremos AGI, e isso, logicamente, é extremamente importante/impactante e precisa ser discutido com cautela.
Pros
- Comparações históricas, como o Projeto Manhattan
- Rigorosidade nos cálculos para todas as estimativas.
- Uso extensivo de fontes.
- Infos sobre uso de energia e crescimento nos próximos anos, principalmente guiadas por necessidade de treino de IAs.
Cons
BEM repetitivo. Da pra deixar 5x menor.
O autor (extremamente qualificado) apresenta muitos fatos para apoiar suas afirmações, mas o ponto que tiro é que, mesmo que ele esteja muito otimista na velocidade do progresso em IA, se assumirmos qualquer taxa de progresso, um dia teremos AGI, e isso, logicamente, é extremamente importante/impactante e precisa ser discutido com cautela.
Pros
- Comparações históricas, como o Projeto Manhattan
- Rigorosidade nos cálculos para todas as estimativas.
- Uso extensivo de fontes.
- Infos sobre uso de energia e crescimento nos próximos anos, principalmente guiadas por necessidade de treino de IAs.
Cons
BEM repetitivo. Da pra deixar 5x menor.
March 22, 2025
Introduced to my friends and I as "the most important book you can read right now". I'm not an expert in either AI or geopolitics, but these were sobering predictions as to how far research has/will come and what impact that will have on the world. The presentation is heady but short and draws comparisons to situations that are easier to grasp for non-experts like me.
I've mostly ignored AI because I've assumed that it can only be as smart as the people that train it and that we shouldn't abandon exercising our God-given intelligence for that of a "smart high schooler" (sometimes I am a snob). Training AI to improve itself is a huge paradigm shift and the implications are massive, especially in the light of CS Lewis' Abolition of Man.
Coming away with a lot to think about. Even when nations are racing for AGI and people are using it to generate their own hell, God will be sovereign. In the meantime, I'll try not to be *that* guy that always talks about AI at social gatherings.
I've mostly ignored AI because I've assumed that it can only be as smart as the people that train it and that we shouldn't abandon exercising our God-given intelligence for that of a "smart high schooler" (sometimes I am a snob). Training AI to improve itself is a huge paradigm shift and the implications are massive, especially in the light of CS Lewis' Abolition of Man.
Coming away with a lot to think about. Even when nations are racing for AGI and people are using it to generate their own hell, God will be sovereign. In the meantime, I'll try not to be *that* guy that always talks about AI at social gatherings.
June 28, 2024
"Situational Awareness" offers an insightful analysis of our proximity to a critical threshold in AI capabilities. His background in machine learning and economics lends credibility to his predictions.
The paper left me with a rather different set of confusions than I started with.
Rapid Progress
Aschenbrenner's extrapolation of recent trends culminates in the onset of an intelligence explosion:
His assessment of GPT-4 as equivalent to a smart high schooler depends significantly on the metrics used. For long-term planning abilities, this estimate may be overstated by about five orders of magnitude. However, by other measures, his assessment seems somewhat reasonable.
Initially, I expected the timeline for automated AI researchers to be slightly longer than his 2028 prediction, due to limitations in their long-term planning abilities. However, upon closer examination, I found his argument less dependent on overcoming such weaknesses than I first thought. So I'm not going to bet very much against his claim here.
While a 3x pace seems somewhat high to me - I'd estimate closer to a 1:1 ratio - his overall forecast for 2028 may not be far off, considering that he may be overestimating the gap between a smart high schooler and an assistant AI researcher.
He has a section on the "data wall" that seems a bit suspicious. He expects increasing divergence in the results of various lab's progress due to need for increasingly important algorithmic insights to get around the problem.
While AI training is indeed data-dependent, and much of the easily accessible data has been used, I believe data scarcity may be less problematic than he suggests. Rather than a "wall," I see it as having picked the low-hanging fruit. Untapped sources of useful data likely exist, with the primary challenge being the cost of acquisition. I'm reluctant to give examples, just in case there are players who haven't figured it out yet. I suspect the most advanced labs will be bottlenecked more by compute than by data.
There will be modest differences in how quickly labs throw lots of money at gathering data. If labs' progress diverges much, it will likely be due to something else (see the next section, on unhobbling).
He decomposes the drivers of progress into three factors: physical compute, algorithmic efficiencies, and unhobbling.
Physical compute is expected to increase nearly four-fold annually until around 2030, with subsequent acceleration or deceleration depending on whether AI has a dramatic transformative impact on global economic growth.
Algorithmic efficiencies, focusing on low-level optimizations, have been surprisingly impactful, doubling effective compute roughly every eight months. I'm guessing this includes minor improvements to matrix multiply algorithms, or figuring out that some operations can be skipped because they don't affect the end result.
The evidence here isn't strong enough to establish a clear long-term trend. My intuition says that it's partly due to a burst of improvements from 2019 to 2022, as researchers suddenly realized those improvements were ridiculously valuable, with diminishing returns potentially already slowing these effects.
Unhobbling
The concept of "unhobbling" suggests that current AIs possess latent human-level intelligence capabilities, hampered by clumsy usage. This potential is being unlocked through high-level algorithmic advances like chain of thought, expanded context windows, and scaffolding.
E.g.:
We're still just beginning to figure out how to turn GPT-4 into a worker that has developed some expertise in a particular job.
His framing of high-level algorithmic progress as unhobbling of latent intelligence is somewhat unusual. Most discussions of AI seem to assume that existing AIs need to acquire some additional source of basic intelligence in order to function at near-human levels.
Is his framing is better? He seems at least partly correct here. When performance is improved by simple tricks such as offering a chatbot a tip, it's pretty clear there's some hidden intelligence that hasn't be fully exposed.
I'm unsure whether most high-level algorithmic progress is better described as unhobbling or as finding new sources of intelligence. The magnitude of such latent intelligence is hard to evaluate. For now, I'm alternating between the unhobbling model and the new sources of intelligence model.
He estimates unhobbling to be as significant as the other two drivers. While evidence is inconclusive, it's conceivable that unhobbling could become the primary driver of progress in the coming years. This uncertainty is makes me nervous.
Intelligence Explosion
I've been somewhat reluctant to use the term intelligence explosion, because it has been associated with a model from Eliezer Yudkowsky that seems somewhat wrong. Aschenbrenner's description of an intelligence explosion aligns more closely with a Hansonian framing. It's more compatible with my understanding of the emergence of human intelligence, and potentially even the Cambrian explosion.
His projection suggests AIs will take over much of AI research by late 2027.
While millions of AI researcher copies is higher than what I expect, the overall analysis doesn't hinge on this specific number.
This model posits that the explosion begins when the cognitive resources devoted to AI development increase at superhuman rates, not necessarily requiring AIs to perform all relevant tasks. Doing AI research requires some specialized brilliance, but doesn't require researchers whose abilities are as general-purpose as humans.
The massive increase in labor could accelerate algorithmic progress by at least 10x - a change dramatic enough to warrant the term "explosion."
I can believe that we'll get a year of 10x algorithmic progress. I expect that after that year, progress will depend much more heavily on compute.
How much will that increase in intelligence enable faster production of compute? He doesn't tackle that question, and I'm fairly uncertain.
It seems ironic that Aschenbrenner has used Hansonian framing to update my beliefs modestly towards Eliezer's prediction of a fast takeoff. Although most of the new evidence provided is about trends in algorithmic progress.
The prediction that LLM-based AI will trigger the explosion doesn't mean that superintelligence will be an LLM:
Superalignment
Aschenbrenner acknowledges significant concerns about safe AI development. However, his tone, particularly in his podcast with Dwarkesh, sounds very much the opposite of scared.
This seems more like fatalism than a well thought out plan. I suspect he finds it hard to imagine scenarios under which safety takes more than a year to develop with AI assistance, so he prays that that will be enough. Or maybe he's seen fear paralyze some leading AI safety advocates, and wants to err in the other direction?
Lock Down the Labs
Aschenbrenner anticipates that competition between the US and China will pressure AI labs to compromise safety.
He recommends military-level security to protect key algorithmic breakthroughs, arguing that this is the primary area where the US can outcompete China.
Such security would slow AI advances, partly by reducing communications within each AI lab, and partly by impairing the ability of AI labs to hire employees who might be blackmailed by the CCP. Presumably he thinks the slowdown is small compared to the difference in how fast the two countries can make algorithmic progress on their own. I'm disturbed that he's not explicit about this. It's not at all obvious that he's correct here.
What about the current US lead in compute? Why won't it be enough for the US to win a race? China may outbuild the US.
I can see how they might build more datacenters than the US. But what chips would they put into them? China now has little access to the best NVIDIA chips or ASML equipment. Those companies have proven hard for anyone to compete with. My impression is that even if China is on track to have twice as many datacenters, they're going to be running at somewhat less than half the speed of US datacenters.
He seems to think that China can make up with quantity for their lack of quality. That's a complex topic. It looks like most experts think he's wrong. But I see few signs of experts who are thinking more deeply than Aschenbrenner about this.
Can we see signs of a massive Chinese datacenter buildup now? My attempts at researching this yielded reports such as this predicting 3.54% annual growth in datacenter construction. That seems ridiculously low even if China decides that AI progress is slowing.
What about stocks of companies involved in the buildup? GDS Holdings and VNET Group seem to be the best available indicators of Chinese datacenter activity. Markets are very much not predicting a boom there. But I suppose the CCP could have serious plans that have been successfully kept secret so far.
My guess is that he is wrong about Chinese ability to catch up to the US in compute by 2028, unless US regulation significantly restricts compute.
What happens when the free world's progress is slowed by chip foundries being destroyed when China invades Taiwan? He is aware that this is somewhat likely to happen this decade, but he says little about what it implies.
China will have some awareness of the possibility of an intelligence explosion. That might influence the timing of military action.
I fear that his anti-CCP attitude will increase the risk of an all-out arms race.
His approach here practically guarantees the kind of arms race that will lead to hasty decisions about whether an AI is safe. His argument seems to be that such a race is nearly inevitable, so the top priority should be ensuring that the better side wins. That could be a self-fulfilling prophecy.
Here's a contrary opinion:
Military Interest
I've been neglecting scenarios under which one or more militaries will take control of AI development, likely because I've been overly influenced by people on LessWrong who expect a brilliant insight to create an intelligence explosion that happens too fast for governments to react.
Aschenbrenner convinced me to expect a somewhat faster intelligence explosion than I previously expected. There's some sense in which that moves me closer to Eliezer's position. But he and I both believe that the intelligence explosion will be a somewhat predictable result of some long-running trends.
So smart people in government are likely realizing now that the military implications deserve careful attention. If the US government is as competent now as it was in the early 1940s, then we'll get something like the Manhattan Project. COVID has created some doubts as to whether people that competent are still in the government. But I suspect the military is more careful than most other parts of government to promote competent people. So I see at least a 50% chance that Aschenbrenner is correct here.
I'm not happy with military involvement. But if it's going to happen, it seems better for it to happen now rather than later. A semi-prepared military is likely to make saner decisions than one that waits to prepare until the intelligence explosion.
The world seems on track for a risky arms race between the free world and China. But I can imagine a sudden shift to a very different trajectory. All it would take is one fire alarm from an AI that's slightly smarter than humans doing something malicious that causes significant alarm. Manifold takes that possibility somewhat seriously.
One key mistake by such an AI could be enough to unite the US and China against the common threat of rogue AI. I don't expect AI at the slightly smarter than human stage to be saner and more rational than humans. It feels scary how much will depend on the details of mistakes made by such AIs.
How Close will the Arms Race be?
I feel more confused about the likely arms race than before.
He suggests treating the arms race as inevitable. Yet his analysis doesn't suggest that the US will maintain much of a lead. He expects military-grade security to be implemented too late to keep the most important algorithmic advances out of CCP hands. I've been assuming a US hardware advantage will cause the US to win a race, but he expects that advantage to disappear. Even worse, the scenario that he predicts seems quite likely to push China to attack Taiwan at a key time, cutting off the main US supply of chips.
Would that mean China pulls ahead? Or that the US is compelled to bomb China's chip factories? These scenarios seem beyond my ability to analyze.
Or if I'm right about China remaining behind in hardware, maybe the attack on Taiwan slows AI progress just as it reaches the intelligence explosion, buying some time at the critical juncture for adequate safety work.
He doesn't appear to have a lot of expertise in this area, but I'm unclear on how to find a better expert.
Conclusion
He has a more credible model than does Eliezer of an intelligence explosion. Don't forget that all models are wrong, but some are useful. Look at the world through multiple models, and don't get overconfident about your ability to pick the best model.
His "optimistic" perspective has increased my gut-level sense of urgency and led me to revise my probability of an existential catastrophe from 12% to 15%, primarily due to the increased likelihood of a closely-fought arms race.
He's overconfident, but well over half right.
The paper contains more valuable insights than I can summarize in one post. It is important reading for anyone interested in the future of AI development and its global implications.
The paper left me with a rather different set of confusions than I started with.
Rapid Progress
Aschenbrenner's extrapolation of recent trends culminates in the onset of an intelligence explosion:
His assessment of GPT-4 as equivalent to a smart high schooler depends significantly on the metrics used. For long-term planning abilities, this estimate may be overstated by about five orders of magnitude. However, by other measures, his assessment seems somewhat reasonable.
Initially, I expected the timeline for automated AI researchers to be slightly longer than his 2028 prediction, due to limitations in their long-term planning abilities. However, upon closer examination, I found his argument less dependent on overcoming such weaknesses than I first thought. So I'm not going to bet very much against his claim here.
One neat way to think about this is that the current trend of AI progress is proceeding at roughly 3x the pace of child development. Your 3x-speed-child just graduated high school; it’ll be taking your job before you know it!
While a 3x pace seems somewhat high to me - I'd estimate closer to a 1:1 ratio - his overall forecast for 2028 may not be far off, considering that he may be overestimating the gap between a smart high schooler and an assistant AI researcher.
He has a section on the "data wall" that seems a bit suspicious. He expects increasing divergence in the results of various lab's progress due to need for increasingly important algorithmic insights to get around the problem.
While AI training is indeed data-dependent, and much of the easily accessible data has been used, I believe data scarcity may be less problematic than he suggests. Rather than a "wall," I see it as having picked the low-hanging fruit. Untapped sources of useful data likely exist, with the primary challenge being the cost of acquisition. I'm reluctant to give examples, just in case there are players who haven't figured it out yet. I suspect the most advanced labs will be bottlenecked more by compute than by data.
There will be modest differences in how quickly labs throw lots of money at gathering data. If labs' progress diverges much, it will likely be due to something else (see the next section, on unhobbling).
He decomposes the drivers of progress into three factors: physical compute, algorithmic efficiencies, and unhobbling.
Physical compute is expected to increase nearly four-fold annually until around 2030, with subsequent acceleration or deceleration depending on whether AI has a dramatic transformative impact on global economic growth.
Algorithmic efficiencies, focusing on low-level optimizations, have been surprisingly impactful, doubling effective compute roughly every eight months. I'm guessing this includes minor improvements to matrix multiply algorithms, or figuring out that some operations can be skipped because they don't affect the end result.
Or consider adaptive compute: Llama 3 still spends as much compute on predicting the “and” token as it does the answer to some complicated question, which seems clearly suboptimal.
The evidence here isn't strong enough to establish a clear long-term trend. My intuition says that it's partly due to a burst of improvements from 2019 to 2022, as researchers suddenly realized those improvements were ridiculously valuable, with diminishing returns potentially already slowing these effects.
Unhobbling
The concept of "unhobbling" suggests that current AIs possess latent human-level intelligence capabilities, hampered by clumsy usage. This potential is being unlocked through high-level algorithmic advances like chain of thought, expanded context windows, and scaffolding.
E.g.:
GPT-4 has the raw smarts to do a decent chunk of many people’s jobs, but it’s sort of like a smart new hire that just showed up 5 minutes ago
We're still just beginning to figure out how to turn GPT-4 into a worker that has developed some expertise in a particular job.
His framing of high-level algorithmic progress as unhobbling of latent intelligence is somewhat unusual. Most discussions of AI seem to assume that existing AIs need to acquire some additional source of basic intelligence in order to function at near-human levels.
Is his framing is better? He seems at least partly correct here. When performance is improved by simple tricks such as offering a chatbot a tip, it's pretty clear there's some hidden intelligence that hasn't be fully exposed.
I'm unsure whether most high-level algorithmic progress is better described as unhobbling or as finding new sources of intelligence. The magnitude of such latent intelligence is hard to evaluate. For now, I'm alternating between the unhobbling model and the new sources of intelligence model.
He estimates unhobbling to be as significant as the other two drivers. While evidence is inconclusive, it's conceivable that unhobbling could become the primary driver of progress in the coming years. This uncertainty is makes me nervous.
Intelligence Explosion
I've been somewhat reluctant to use the term intelligence explosion, because it has been associated with a model from Eliezer Yudkowsky that seems somewhat wrong. Aschenbrenner's description of an intelligence explosion aligns more closely with a Hansonian framing. It's more compatible with my understanding of the emergence of human intelligence, and potentially even the Cambrian explosion.
His projection suggests AIs will take over much of AI research by late 2027.
We’d be able to run millions of copies (and soon at 10x+ human speed) of the automated AI researchers.
While millions of AI researcher copies is higher than what I expect, the overall analysis doesn't hinge on this specific number.
Imagine 1000 automated AI researchers spending a month-equivalent checking your code and getting the exact experiment right before you press go. I’ve asked some AI lab colleagues about this and they agreed: you should pretty easily be able to save 3x-10x of compute on most projects merely if you could avoid frivolous bugs, get things right on the first try, and only run high value-of-information experiments.
This model posits that the explosion begins when the cognitive resources devoted to AI development increase at superhuman rates, not necessarily requiring AIs to perform all relevant tasks. Doing AI research requires some specialized brilliance, but doesn't require researchers whose abilities are as general-purpose as humans.
The massive increase in labor could accelerate algorithmic progress by at least 10x - a change dramatic enough to warrant the term "explosion."
I can believe that we'll get a year of 10x algorithmic progress. I expect that after that year, progress will depend much more heavily on compute.
How much will that increase in intelligence enable faster production of compute? He doesn't tackle that question, and I'm fairly uncertain.
It seems ironic that Aschenbrenner has used Hansonian framing to update my beliefs modestly towards Eliezer's prediction of a fast takeoff. Although most of the new evidence provided is about trends in algorithmic progress.
The prediction that LLM-based AI will trigger the explosion doesn't mean that superintelligence will be an LLM:
The superintelligence we get by the end of it could be quite alien. We’ll have gone through a decade or more of ML advances during the intelligence explosion, meaning the architectures and training algorithms will be totally different (with potentially much riskier safety properties).
Superalignment
RLHF relies on humans being able to understand and supervise AI behavior, which fundamentally won’t scale to superhuman systems.
By default, it may well learn to lie, to commit fraud, to deceive, to hack, to seek power, and so on
The primary problem is that for whatever you want to instill the model (including ensuring very basic things, like “follow the law”!) we don’t yet know how to do that for the very powerful AI systems we are building very soon.
... What’s more, I expect that within a small number of years, these AI systems will be integrated in many critical systems, including military systems (failure to do so would mean complete dominance by adversaries).
Aschenbrenner acknowledges significant concerns about safe AI development. However, his tone, particularly in his podcast with Dwarkesh, sounds very much the opposite of scared.
This seems more like fatalism than a well thought out plan. I suspect he finds it hard to imagine scenarios under which safety takes more than a year to develop with AI assistance, so he prays that that will be enough. Or maybe he's seen fear paralyze some leading AI safety advocates, and wants to err in the other direction?
Lock Down the Labs
Aschenbrenner anticipates that competition between the US and China will pressure AI labs to compromise safety.
in the next 12-24 months, we will develop the key algorithmic breakthroughs for AGI, and promptly leak them to the CCP
But the AI labs are developing the algorithmic secrets—the key technical breakthroughs, the blueprints so to speak—for the AGI right now (in particular, the RL/self-play/synthetic data/etc “next paradigm” after LLMs to get past the data wall). AGI-level security for algorithmic secrets is necessary years before AGI-level security for weights. These algorithmic breakthroughs will matter more than a 10x or 100x larger cluster in a few years
a healthy lead will be the necessary buffer that gives us margin to get AI safety right, too ... the difference between a 1-2 year and 1-2 month lead will really matter for navigating the perils of superintelligence
He recommends military-level security to protect key algorithmic breakthroughs, arguing that this is the primary area where the US can outcompete China.
Such security would slow AI advances, partly by reducing communications within each AI lab, and partly by impairing the ability of AI labs to hire employees who might be blackmailed by the CCP. Presumably he thinks the slowdown is small compared to the difference in how fast the two countries can make algorithmic progress on their own. I'm disturbed that he's not explicit about this. It's not at all obvious that he's correct here.
What about the current US lead in compute? Why won't it be enough for the US to win a race? China may outbuild the US.
The binding constraint on the largest training clusters won’t be chips, but industrial mobilization—perhaps most of all the 100GW of power for the trillion-dollar cluster. But if there’s one thing China can do better than the US it’s building stuff.
I can see how they might build more datacenters than the US. But what chips would they put into them? China now has little access to the best NVIDIA chips or ASML equipment. Those companies have proven hard for anyone to compete with. My impression is that even if China is on track to have twice as many datacenters, they're going to be running at somewhat less than half the speed of US datacenters.
He seems to think that China can make up with quantity for their lack of quality. That's a complex topic. It looks like most experts think he's wrong. But I see few signs of experts who are thinking more deeply than Aschenbrenner about this.
Can we see signs of a massive Chinese datacenter buildup now? My attempts at researching this yielded reports such as this predicting 3.54% annual growth in datacenter construction. That seems ridiculously low even if China decides that AI progress is slowing.
What about stocks of companies involved in the buildup? GDS Holdings and VNET Group seem to be the best available indicators of Chinese datacenter activity. Markets are very much not predicting a boom there. But I suppose the CCP could have serious plans that have been successfully kept secret so far.
My guess is that he is wrong about Chinese ability to catch up to the US in compute by 2028, unless US regulation significantly restricts compute.
What happens when the free world's progress is slowed by chip foundries being destroyed when China invades Taiwan? He is aware that this is somewhat likely to happen this decade, but he says little about what it implies.
China will have some awareness of the possibility of an intelligence explosion. That might influence the timing of military action.
I fear that his anti-CCP attitude will increase the risk of an all-out arms race.
His approach here practically guarantees the kind of arms race that will lead to hasty decisions about whether an AI is safe. His argument seems to be that such a race is nearly inevitable, so the top priority should be ensuring that the better side wins. That could be a self-fulfilling prophecy.
Here's a contrary opinion:
What US/China AI race folk sound like to me:
There are superintelligent super technologically advanced aliens coming towards earth at .5 C. We don't know anything about their values. The most important thing to do is make sure they land in the US before they land in China.
Military Interest
The intelligence explosion will be more like running a war than launching a product. ... I find it an insane proposition that the US government will let a random SF startup develop superintelligence. Imagine if we had developed atomic bombs by letting Uber just improvise.
I've been neglecting scenarios under which one or more militaries will take control of AI development, likely because I've been overly influenced by people on LessWrong who expect a brilliant insight to create an intelligence explosion that happens too fast for governments to react.
Aschenbrenner convinced me to expect a somewhat faster intelligence explosion than I previously expected. There's some sense in which that moves me closer to Eliezer's position. But he and I both believe that the intelligence explosion will be a somewhat predictable result of some long-running trends.
So smart people in government are likely realizing now that the military implications deserve careful attention. If the US government is as competent now as it was in the early 1940s, then we'll get something like the Manhattan Project. COVID has created some doubts as to whether people that competent are still in the government. But I suspect the military is more careful than most other parts of government to promote competent people. So I see at least a 50% chance that Aschenbrenner is correct here.
I'm not happy with military involvement. But if it's going to happen, it seems better for it to happen now rather than later. A semi-prepared military is likely to make saner decisions than one that waits to prepare until the intelligence explosion.
It seems pretty clear: this should not be under the unilateral command of a random CEO. Indeed, in the private-labs-developing-superintelligence world, it’s quite plausible individual CEOs would have the power to literally coup the US government.
The world seems on track for a risky arms race between the free world and China. But I can imagine a sudden shift to a very different trajectory. All it would take is one fire alarm from an AI that's slightly smarter than humans doing something malicious that causes significant alarm. Manifold takes that possibility somewhat seriously.
One key mistake by such an AI could be enough to unite the US and China against the common threat of rogue AI. I don't expect AI at the slightly smarter than human stage to be saner and more rational than humans. It feels scary how much will depend on the details of mistakes made by such AIs.
How Close will the Arms Race be?
I feel more confused about the likely arms race than before.
He suggests treating the arms race as inevitable. Yet his analysis doesn't suggest that the US will maintain much of a lead. He expects military-grade security to be implemented too late to keep the most important algorithmic advances out of CCP hands. I've been assuming a US hardware advantage will cause the US to win a race, but he expects that advantage to disappear. Even worse, the scenario that he predicts seems quite likely to push China to attack Taiwan at a key time, cutting off the main US supply of chips.
Would that mean China pulls ahead? Or that the US is compelled to bomb China's chip factories? These scenarios seem beyond my ability to analyze.
Or if I'm right about China remaining behind in hardware, maybe the attack on Taiwan slows AI progress just as it reaches the intelligence explosion, buying some time at the critical juncture for adequate safety work.
He doesn't appear to have a lot of expertise in this area, but I'm unclear on how to find a better expert.
Conclusion
He has a more credible model than does Eliezer of an intelligence explosion. Don't forget that all models are wrong, but some are useful. Look at the world through multiple models, and don't get overconfident about your ability to pick the best model.
His "optimistic" perspective has increased my gut-level sense of urgency and led me to revise my probability of an existential catastrophe from 12% to 15%, primarily due to the increased likelihood of a closely-fought arms race.
He's overconfident, but well over half right.
The paper contains more valuable insights than I can summarize in one post. It is important reading for anyone interested in the future of AI development and its global implications.
December 18, 2024
Scared of the future :(
August 4, 2024
Very interesting essay … paints a picture that seems increasingly likely.
September 19, 2024
3 stars for this readable and informative review (no, I don't think it's a book). Although I think the author is too optimistic about training sources and material science-related hardware updates. Currently, the computational biology/chemistry/materials are majorly used to explain but not predict the structure-function relation, so I think this bottleneck is not easy to breakthrough even with AGI.
1 star off is because I dislike the cold-war mindset that the author is passionate about. He said something like "our generation is too comfortable without the threatening of war..." before he started to preach his anti-globalization and "American first" idea. I mean... even he doesn't like China (and he has not even mentioned Russia), does he truly so naively think the democratic allies in Europe, Japan and S Korea really like this passive, nationalistic, America-centered speech? The fall of Russian technology was partially contributed by the lacking in PC and Internet development. The failure of catching the PC and internet tide of Russia originated from their isolated and SPECIFIC-AIM focused research plan, while the collaborative, self-interest-oriented and capitalism-driven strategies made SF in the US prospering over the years. Although the author speak for "democracy", but clearly he really likes the planning style of Soviet government.
1 star off is because I dislike the cold-war mindset that the author is passionate about. He said something like "our generation is too comfortable without the threatening of war..." before he started to preach his anti-globalization and "American first" idea. I mean... even he doesn't like China (and he has not even mentioned Russia), does he truly so naively think the democratic allies in Europe, Japan and S Korea really like this passive, nationalistic, America-centered speech? The fall of Russian technology was partially contributed by the lacking in PC and Internet development. The failure of catching the PC and internet tide of Russia originated from their isolated and SPECIFIC-AIM focused research plan, while the collaborative, self-interest-oriented and capitalism-driven strategies made SF in the US prospering over the years. Although the author speak for "democracy", but clearly he really likes the planning style of Soviet government.
July 4, 2024
First, if you want some serious headaches, nightmares, and worries about the end of the world in the near future; then read this report and read it uncritically.
This is a simplified, popular, desperate (not so say paranoiac), political, and urgent version of Bostrom's “Superintelligence” - following the release of GPT-4-like models, plus all kinds of catastrophic AI prospects during the next decade. Basically - the author of this free and web-available report sees the current situation in AI similar to the one right before the Manhattan Project started: with AGI instead of the atomic bomb, with him in the role of Leo Szilard, with the Chinese in the role of the Germans/Russians, with the inevitable and massive Project initiated and controlled by the US government, with the San Francisco's AI community as the equivalent of the atomic physicists, with a lot of spies, with this report as the equivalent of the letter that Szilard/Einstein wrote to the US president, with massive geopolitical, military, and economic shifts, so on.
The best part of this report is its first sentence: “You can see the future first in San Francisco”. Great also are some details about trends, costs, and other insights that only someone working in the AI industry knows.
The author believes that we are on a course to have AGI by 2027; and immediately after that will follow an intelligence explosion and of course superintelligence. At this latter point we completely lost control of the superintelligence – for better, but most likely for worse. However, on the first page of this report he writes: “Nvidia analysts still think 2024 might be close to the peak. Mainstream pundits are stuck on the willful blindness of 'it’s just predicting the next word'. They see only hype and business-as-usual; at most they entertain another internet-scale technological change.” My personal bets are with Nvidia and their professional projections - a company that so far was the big winner from all this AI craziness; and accordingly, I will take 2024 as the peak and not as the beginning of an accelerating AI curve. But we will need to wait and see on this one.
When compared with the atomic bomb project - the science was already there at that time, and only the details needed to be figured out. These days, it seems to me that no one is understanding how to move from the current GPT-4 or similar to AGI. That is, everyone is hoping that somehow AGI will spontaneously emerge when enough CPU power is mobilized – along with the associated massive amounts of energy, experts, and algorithms that will help in the training of the “trillion dollars cluster” (i.e., this cluster is the main pivotal point in this report when the transition from AI to AGI will happen, and according to the author it will arrive in the next few years). Here are two statements from this report that shows that GPT-4 was completely unexpected and that no one understands it (i.e. that there is no science behind AGI and thus not really a path forward): “GPT-4’s capabilities came as a shock to many” and “by default, modern AI systems are inscrutable black boxes.”
One of my main questions here is how the superintelligence will develop some form of conscience or self-autonomy in order to start doing things by itself. So far, none of the GPT-like models showed any initiative and conscience. To me it is almost sure that the new and more complex models will show none also. In fact - there is no theory of conscience or selfness/I there; not to mention any such theory that can be implemented on a computer or one generated by logical processes.
It seems to me that here - as in all similar books or arguments - intelligence's definition is taken as obvious and for granted. All we have in this report are some intelligence categories: “preschooler”, “elementary schooler”, “smart high schooler”, “university”, “PhD”, and similar . Moreover, the models are not smarter or more intelligent as they move from preschooler to PhD, but simply they are trained on PhD data and not on preschooler data. If so and as an example, to have a model that understands quantum physics at PhD level all you need is a lot of such data to train it and to set-up a model complex enough to accommodate it. These free-floating notions of intelligence/smartness that automatically are self-sustaining and produce more and more intelligence, research, and all kinds of new or fundamental information does not make any sense to me – except if you take the entire enlightenment project that defined reason/intelligence in a very naive and superficial way.
This training issue brings another issue discussed here at length and acknowledged as a major one – the data wall. It seems that the current modes are already using all the data on the Internet plus all non-free digital data out there. Unless a solution to this “data wall” will be found, most likely the future models will not be much better than the current ones. The hope here is that these models will produce new research, science, and new information that will be imputed back into the more advanced/next models – but so far all these models are just parrots or “high-tech plagiarism”. Also, the dream of some adversarial models - like the ones developed by DeepMind to play Statecraft and AlphaGo against themselves and thus to generate new data - do not seem to apply here at all.
As I see it, there is a business side and a prophetic side of this AI-AGI-Superintelligence issue. The business side struck gold recently and they are mining furiously and investing massively for more returns. The prophetic side is quite deep (i.e., in a Jungian sense) and is simply replacing religious notions of immortality, God, paradise, hell, and so on - that the scientific worldview denounces as baseless over the last few hundred years - with new ones that are man-made but not much different from the old ones. The prophets see in the current business boom a mystical emergence that will accelerate to the man-made God/Singularity in 3-4 years – and it is not clear to them if this will end with the Apocalypse or some form of heaven. As mentioned above, we will see how Nvidia stock will move during the next 1-2 years - and then we will know for sure if all this is just the third AI bubble about to burst or the beginning of the Apocalypse.
This is a simplified, popular, desperate (not so say paranoiac), political, and urgent version of Bostrom's “Superintelligence” - following the release of GPT-4-like models, plus all kinds of catastrophic AI prospects during the next decade. Basically - the author of this free and web-available report sees the current situation in AI similar to the one right before the Manhattan Project started: with AGI instead of the atomic bomb, with him in the role of Leo Szilard, with the Chinese in the role of the Germans/Russians, with the inevitable and massive Project initiated and controlled by the US government, with the San Francisco's AI community as the equivalent of the atomic physicists, with a lot of spies, with this report as the equivalent of the letter that Szilard/Einstein wrote to the US president, with massive geopolitical, military, and economic shifts, so on.
The best part of this report is its first sentence: “You can see the future first in San Francisco”. Great also are some details about trends, costs, and other insights that only someone working in the AI industry knows.
The author believes that we are on a course to have AGI by 2027; and immediately after that will follow an intelligence explosion and of course superintelligence. At this latter point we completely lost control of the superintelligence – for better, but most likely for worse. However, on the first page of this report he writes: “Nvidia analysts still think 2024 might be close to the peak. Mainstream pundits are stuck on the willful blindness of 'it’s just predicting the next word'. They see only hype and business-as-usual; at most they entertain another internet-scale technological change.” My personal bets are with Nvidia and their professional projections - a company that so far was the big winner from all this AI craziness; and accordingly, I will take 2024 as the peak and not as the beginning of an accelerating AI curve. But we will need to wait and see on this one.
When compared with the atomic bomb project - the science was already there at that time, and only the details needed to be figured out. These days, it seems to me that no one is understanding how to move from the current GPT-4 or similar to AGI. That is, everyone is hoping that somehow AGI will spontaneously emerge when enough CPU power is mobilized – along with the associated massive amounts of energy, experts, and algorithms that will help in the training of the “trillion dollars cluster” (i.e., this cluster is the main pivotal point in this report when the transition from AI to AGI will happen, and according to the author it will arrive in the next few years). Here are two statements from this report that shows that GPT-4 was completely unexpected and that no one understands it (i.e. that there is no science behind AGI and thus not really a path forward): “GPT-4’s capabilities came as a shock to many” and “by default, modern AI systems are inscrutable black boxes.”
One of my main questions here is how the superintelligence will develop some form of conscience or self-autonomy in order to start doing things by itself. So far, none of the GPT-like models showed any initiative and conscience. To me it is almost sure that the new and more complex models will show none also. In fact - there is no theory of conscience or selfness/I there; not to mention any such theory that can be implemented on a computer or one generated by logical processes.
It seems to me that here - as in all similar books or arguments - intelligence's definition is taken as obvious and for granted. All we have in this report are some intelligence categories: “preschooler”, “elementary schooler”, “smart high schooler”, “university”, “PhD”, and similar . Moreover, the models are not smarter or more intelligent as they move from preschooler to PhD, but simply they are trained on PhD data and not on preschooler data. If so and as an example, to have a model that understands quantum physics at PhD level all you need is a lot of such data to train it and to set-up a model complex enough to accommodate it. These free-floating notions of intelligence/smartness that automatically are self-sustaining and produce more and more intelligence, research, and all kinds of new or fundamental information does not make any sense to me – except if you take the entire enlightenment project that defined reason/intelligence in a very naive and superficial way.
This training issue brings another issue discussed here at length and acknowledged as a major one – the data wall. It seems that the current modes are already using all the data on the Internet plus all non-free digital data out there. Unless a solution to this “data wall” will be found, most likely the future models will not be much better than the current ones. The hope here is that these models will produce new research, science, and new information that will be imputed back into the more advanced/next models – but so far all these models are just parrots or “high-tech plagiarism”. Also, the dream of some adversarial models - like the ones developed by DeepMind to play Statecraft and AlphaGo against themselves and thus to generate new data - do not seem to apply here at all.
As I see it, there is a business side and a prophetic side of this AI-AGI-Superintelligence issue. The business side struck gold recently and they are mining furiously and investing massively for more returns. The prophetic side is quite deep (i.e., in a Jungian sense) and is simply replacing religious notions of immortality, God, paradise, hell, and so on - that the scientific worldview denounces as baseless over the last few hundred years - with new ones that are man-made but not much different from the old ones. The prophets see in the current business boom a mystical emergence that will accelerate to the man-made God/Singularity in 3-4 years – and it is not clear to them if this will end with the Apocalypse or some form of heaven. As mentioned above, we will see how Nvidia stock will move during the next 1-2 years - and then we will know for sure if all this is just the third AI bubble about to burst or the beginning of the Apocalypse.
April 13, 2025
Briefly fell into some AI fear and mania when I first finished, but after touching grass I identified some core issues with this paper / book that I've listed below. Overall, this is a great discussion piece for a book club but I don't think it is grounded in reality -- let's see if in a few years I eat my words! Clock is ticking given the time horizon presented...
1) The graphs throughout are laughable / why did he bother showing them? Cracked up at the chart with the X-axis representing intelligence, and pre-schooler on the LHS and 'AI Researcher' on the RHS. Think his point was that it reaches the point of a flywheel of AI development, but seems that he is just placing himself as the smartest person in the world. Lots of linear charts that seem a bit nonsensical.
2) Looking into Leopold's background makes me highly skeptical. Running an AI investment firm now... this reads like a shameless shill. He's not necessarily that technical either from the background I can find.
3) His revenue / economic projections were laughable - and given this is the one area covered I somewhat understand, it made me dubious about all the AI advancement arguments. 30%+ GDP growth because of robot mega factories in Nevada? But superintelligence replaced all corporate workers? So many questions.
4) The electricity required is going to come from Pennsylvania natural gas? 20% of today's total US energy consumption earmarked for AI? All by 2027, hmm.
5) Manhattan Project comparison was interesting, but overdone. Leopold seems to really want to be Leo Szilard and be invited to a future government-led AI project.
Will stop listing... I struggled with the first few essays and wasn't convinced on the feasibility of AGI by 2027 or even in the next 10 years. Seems he just threw around 'OOM' with a bunch of silly charts to make his points. Everything else in the paper was a study of how we get there from an infra perspective and how we react to eventual superintelligence, which I found interesting but I had been left behind.
One last point, there is absolutely no discussion of how intelligence is defined. Philosophically I think this is the most interesting consideration, and somewhat essential to the conversation. Where is the indication that AGI would have the spark to make choices and become sentient? Maybe I'm just behind on existing literature, but seems like a gaping hole in this discussion.
1) The graphs throughout are laughable / why did he bother showing them? Cracked up at the chart with the X-axis representing intelligence, and pre-schooler on the LHS and 'AI Researcher' on the RHS. Think his point was that it reaches the point of a flywheel of AI development, but seems that he is just placing himself as the smartest person in the world. Lots of linear charts that seem a bit nonsensical.
2) Looking into Leopold's background makes me highly skeptical. Running an AI investment firm now... this reads like a shameless shill. He's not necessarily that technical either from the background I can find.
3) His revenue / economic projections were laughable - and given this is the one area covered I somewhat understand, it made me dubious about all the AI advancement arguments. 30%+ GDP growth because of robot mega factories in Nevada? But superintelligence replaced all corporate workers? So many questions.
4) The electricity required is going to come from Pennsylvania natural gas? 20% of today's total US energy consumption earmarked for AI? All by 2027, hmm.
5) Manhattan Project comparison was interesting, but overdone. Leopold seems to really want to be Leo Szilard and be invited to a future government-led AI project.
Will stop listing... I struggled with the first few essays and wasn't convinced on the feasibility of AGI by 2027 or even in the next 10 years. Seems he just threw around 'OOM' with a bunch of silly charts to make his points. Everything else in the paper was a study of how we get there from an infra perspective and how we react to eventual superintelligence, which I found interesting but I had been left behind.
One last point, there is absolutely no discussion of how intelligence is defined. Philosophically I think this is the most interesting consideration, and somewhat essential to the conversation. Where is the indication that AGI would have the spark to make choices and become sentient? Maybe I'm just behind on existing literature, but seems like a gaping hole in this discussion.
June 24, 2024
we're calling position papers books now? interesting. i think it might be impossible to read this as anything other than hysterical, but i'm no expert on either the philosophy nor science of AGI. suffice to say i have my doubts regarding superintelligence, but i have no doubts regarding AGI itself, nor that it will be realized soon. a lot of Aschenbrenner's arguments rely on an assumption of superintelligence, especially his most dire assertions. with that said, the answers regarding superintelligence are not at all decided, and if i'm wrong, what he says is probably valid (though i think the resulting changes would go far beyond anything he even hints at, and explode the view through which he analyzes the near future--in the presence of superintelligence, i don't see any need for humanity and certainly not 18th century-style nationstates).
most of his suggestions seem ridiculous. this isn't nuclear weaponry, where you need a uranium/plutonium base requiring a huge industrial effort [0] to get anywhere. *that's* what keeps the Red Hat Ladies from becoming nuclear powers, not secrecy. the idea that innovations in AI, spread across hundreds if not thousands of labs and literally tens of thousands of people, are going to be kept bottled up by any scheme is asinine, and frankly makes me question the author's sensibilities.
everyone ought read this extraordinary document IMHO, but don't necessarily buy its assertions nor conclusions. be aware that it's fairly repetitive, and reads like four presentations stapled together.
my tipoff came from scott aaronson's blog, though this has been all over the CS world for a week or so.
[0] obligatory reference to my novel midnight's simulacra for a potential way around this
most of his suggestions seem ridiculous. this isn't nuclear weaponry, where you need a uranium/plutonium base requiring a huge industrial effort [0] to get anywhere. *that's* what keeps the Red Hat Ladies from becoming nuclear powers, not secrecy. the idea that innovations in AI, spread across hundreds if not thousands of labs and literally tens of thousands of people, are going to be kept bottled up by any scheme is asinine, and frankly makes me question the author's sensibilities.
everyone ought read this extraordinary document IMHO, but don't necessarily buy its assertions nor conclusions. be aware that it's fairly repetitive, and reads like four presentations stapled together.
my tipoff came from scott aaronson's blog, though this has been all over the CS world for a week or so.
[0] obligatory reference to my novel midnight's simulacra for a potential way around this
October 12, 2024
Useful for understanding why AGI needs to be in the hands of the the US government. Two reasons:
- Private development of AI will always lack robust security, i.e., it’s too easy to steal algorithms/weights via convos in the Bay
- AGI is a powerful weapon of mass destruction. If it becomes misaligned (ex. reinforced to lie or automates its own acceleration), it will carry the same weight as nuclear weapons in their threat to human existence.
I’m less convinced by the argument that we’re close to AGI. Aschenbrenner uses historical examples of government-enabled industrial investment to suggest that the US government might act similarly for compute, but he also notes that the US government hasn’t had impetus to properly safeguard American dominance of AGI. He briefly mentions politicization as another concern (ex. climate issues discouraging environmentally unfriendly compute production); my personal opinion is that he underestimates the the effect to which politics might hinder the US government’s ability to centralize and execute on securing AGI.
Good for generally getting up to date with AI with the occassional distracting Gen Z term.
- Private development of AI will always lack robust security, i.e., it’s too easy to steal algorithms/weights via convos in the Bay
- AGI is a powerful weapon of mass destruction. If it becomes misaligned (ex. reinforced to lie or automates its own acceleration), it will carry the same weight as nuclear weapons in their threat to human existence.
I’m less convinced by the argument that we’re close to AGI. Aschenbrenner uses historical examples of government-enabled industrial investment to suggest that the US government might act similarly for compute, but he also notes that the US government hasn’t had impetus to properly safeguard American dominance of AGI. He briefly mentions politicization as another concern (ex. climate issues discouraging environmentally unfriendly compute production); my personal opinion is that he underestimates the the effect to which politics might hinder the US government’s ability to centralize and execute on securing AGI.
Good for generally getting up to date with AI with the occassional distracting Gen Z term.
Read
December 22, 2025Author is very young!
Why so many people agi-pilled from OpenAI?
For the outside, we are just saving our heads at the hallucinations of copilot 5, Gemini 3, etc…, and unclear how much smarter these versions are (at a practical level) compared to the earlier versions that this guy was seeing and freaking out over.
This has some well explained, easy to digest explanations how these robots wombat smarter and actually become useful enough to replace us. He seems to think it’s a physical certainty that asi is possible (with frequent comparisons to manhattan project). Again, I’m j the outside here, and wonder if, as long as the base is a machine learning algorithm, you just gonna eventually overfit and the models get dumber.
But who knows! Right now Claude “agent” is getting a lot of hype and seems like it’s the “agent” that these kool -aid drinkers have been predicting.
Hard to say, but anyways. Interesting read. Im hoping for a workless future, but everybody is predicted arms race and terminator 2 and stuff like that. Bummer!
Why so many people agi-pilled from OpenAI?
For the outside, we are just saving our heads at the hallucinations of copilot 5, Gemini 3, etc…, and unclear how much smarter these versions are (at a practical level) compared to the earlier versions that this guy was seeing and freaking out over.
This has some well explained, easy to digest explanations how these robots wombat smarter and actually become useful enough to replace us. He seems to think it’s a physical certainty that asi is possible (with frequent comparisons to manhattan project). Again, I’m j the outside here, and wonder if, as long as the base is a machine learning algorithm, you just gonna eventually overfit and the models get dumber.
But who knows! Right now Claude “agent” is getting a lot of hype and seems like it’s the “agent” that these kool -aid drinkers have been predicting.
Hard to say, but anyways. Interesting read. Im hoping for a workless future, but everybody is predicted arms race and terminator 2 and stuff like that. Bummer!
This entire review has been hidden because of spoilers.
February 11, 2025
Recommended by Manny. Available for free online, this relatively short 150-page document presents the author’s view that the coming years will be pivotal in the race for AGI (artificial general intelligence) and ASI (artificial superintelligence). Key concerns include AI safety, international competition—particularly with China—and the need for a centralized, government-led effort by 2027 to fund and secure AI research. While the timeline may feel overly optimistic, the work is an engaging and thought-provoking read. I broadly agree with its main arguments, though only time will tell if its predictions hold true.
★★★★☆
★★★★☆
April 27, 2025
I didn’t understand why it took me >6 months to finish this book but probably because Leopold talked for 4 hours on Dwarkesh’s podcast.
It turned out this book isn’t about AI models becoming more situationally aware. It is about humans. I got fully convinced ~80% that the urgency of national security on AGI is crucial, but I wasn’t convinced of handling all these future power to USG. The USG right now might come close to the totalitarianism of CCP.
But the world has gone mad these days. I might change my opinion once I finish AI 2027.
It turned out this book isn’t about AI models becoming more situationally aware. It is about humans. I got fully convinced ~80% that the urgency of national security on AGI is crucial, but I wasn’t convinced of handling all these future power to USG. The USG right now might come close to the totalitarianism of CCP.
But the world has gone mad these days. I might change my opinion once I finish AI 2027.
June 17, 2024
This essay series makes AI safety concerns feel a lot more real. Sure, there are a lot of assumptions included that could be prove wrong, but it doesn't actually seem that far-fetched to be dealing with scarily powerful tech only a few years from now. I hope this spurs more immediate awareness and effective actions to make sure that we are proactively preparing ourselves and society for this technology.
September 1, 2024
ChatGPT focused the minds of AI doomers, but according to the author, there are still very few people who understand the magnitude of the coming change. Aschenbrenner, who used to work for OpenAI, predicts a rapid rise in the AI capacities based on the improvements in data, computing, and algorithms. If the capacities continue to develop at the current pace of an order of magnitude (or somewhat higher), then very quickly AI will surpass human capabilities. The investment in huge clusters is already materialising and those that went long on AI/chip makers made a killing.
What is more disturbing is the dual usage of this technology. Aschenbrenner compares AI to the nuclear bomb, and proposes closing of the technology as otherwise the most powerful weapon in the 21st century will soon get in the hands of its adversaries. In the longer run (i.e. this decade), he believes that the US should create an AI version of the Manhattan Project to achieve supremacy in this technology. The endgame could be an AI wil superhuman abilities that can be (semi)autonomous, and speed up the scientific discovery process. This speed up could be used to improve the efficiency of a military therefore he sees it as vital to keep the latest advances in AI secret. Based on the current threat level (i.e. relatively low), the labs are not taking even this seriously. However, when state actors realise what is at stake, they will deploy their full force to obtain AI secrets.
This is a highly insightful book and is available on the author’s page.
What is more disturbing is the dual usage of this technology. Aschenbrenner compares AI to the nuclear bomb, and proposes closing of the technology as otherwise the most powerful weapon in the 21st century will soon get in the hands of its adversaries. In the longer run (i.e. this decade), he believes that the US should create an AI version of the Manhattan Project to achieve supremacy in this technology. The endgame could be an AI wil superhuman abilities that can be (semi)autonomous, and speed up the scientific discovery process. This speed up could be used to improve the efficiency of a military therefore he sees it as vital to keep the latest advances in AI secret. Based on the current threat level (i.e. relatively low), the labs are not taking even this seriously. However, when state actors realise what is at stake, they will deploy their full force to obtain AI secrets.
This is a highly insightful book and is available on the author’s page.
June 21, 2024
Recently developed an interest in AI and how is it going to affect the rest of our lives so came across to the author on Twitter. This series of writings focuses more on why the US should lead the soon-to-come superintelligence, how it is so important for military advantage, and that the US should start the 'AI Manhattan Project' asap. Therefore, I can't it was what I was hoping it was to read.
August 4, 2024
A both incredibly optimistic (AI is real and is change the world) and pessimistic (AI is the new nuclear bomb and we better control it). Easily readable for a non AI person such as I, so that's a good.
January 19, 2025
I don't think the author is necessarily wrong, I think we do need to be more diligent about our thinking about AI, but I think his timetable is overly optimistic and misses the broader context, including human nature outside of technology.
January 27, 2025
The most annoying part is the insanely naive America-centric view. The great battle for civilization: US as the free world vs bad China. Also: numbers and estimates are just invented out of nothing. Not a serious report
June 18, 2024
Scott Aaronson's recommendation.
August 14, 2024
definitely worth reading
August 24, 2024
3.5
Only an essay but 150 pages so im counting it idgaf. Interesting ideas, clearly very bright and written solidly but glazes over some important ideas
Only an essay but 150 pages so im counting it idgaf. Interesting ideas, clearly very bright and written solidly but glazes over some important ideas
Displaying 1 - 30 of 40 reviews