Status Updates From Situational Awareness: The ...

I just started but this guy is a genius. Nothing less.

These are great and honorable people. But they are just people. Soon, the AIs will be running the world, but we’re in for
one last rodeo. May their final stewardship bring honor to
mankind.

In particular, we may want to “spend some of our lead”
to have time to solve safety challenges, but Western labs will
need to coordinate to do so. (And of course, private labs will
have already had their AGI weights stolen, so their safety pre-
situational awareness 151
cautions won’t even matter; we’ll be at the mercy of the CCP’s
and North Korea’s safety precautions.)

its funny when he says "random nonprofit board" like why not just say OpenAI lol

It is a delusion of those who have unconsciously internalized
our brief respite from history that this will not summon more
primordial forces. Like many scientists before us, the great
minds of San Francisco hope that they can control the destiny
of the demon they are birthing. Right now, they still can; for
they are among the few with situational awareness, who understand what they are building.

There’s already an eerie convergence of AGI timelines (~2027?)
and Taiwan watchers’ Taiwan invasion timelines (China ready
to invade Taiwan by 2027?)—a convergence that will surely
only heighten as the world wakes up to AGI. (Imagine if in
1960, the vast majority of the world’s uranium deposits were
somehow concentrated in Berlin.)

America’s lead on AGI won’t secure peace and freedom by just
building the best AI girlfriend apps. It’s not pretty—but we
must build AI for American defense.

Some hope for some sort of international treaty on safety. This
seems fanciful to me. (How have those climate treaties
gone? That seems like a dramatically easier problem compared
to this.)

To date, US tech companies have made a much bigger bet on
AI and scaling than any Chinese efforts; consequently, we are
well ahead. But counting out China now is a bit like counting out Google in the AI race when ChatGPT came out in late
2022. Google hadn’t yet focused their efforts in an intense AI
bet, and it looked as though OpenAI was far ahead—but

But if there’s one thing China can do better than the US
it’s building stuff.

We’re not on track for superdefense, for
an airgapped cluster or any of that; I’m not sure we would
even realize if a model self-exfiltrated.

For example, if we deliberately plant backdoors
or misalignments into models, would our safety training would
have caught and gotten rid of them? (Early work suggests that
“sleeper agents” can survive through safety training, for example.)

If we do rapidly
transition from from AGI to superintelligence,

typo

the story of Szilard, Fermi and Bothe is amazing.

I sometimes joke that AI lab algorithmic advances are not shared with
the American research community, but
they are being shared with the Chinese
research community!

Google DeepMind Frontier Safety Framework outlines security levels 0, 1, 2, 3, and 4 (~1.5 being what you’d
need to defend against well-resourced terrorist groups or cybercriminals, 3 being what you’d need to defend against the
North Koreas of the world, and 4 being what you’d need to
have even a shot of defending against priority efforts by the
most capable state actors).72 They admit to being at level 0.

Too many smart people underrate espionage.

AGI-level
security for algorithmic secrets is necessary years before AGI level security for weights.

If American business is unshackled, America can build like none other (at least in red states).