What If AI Actually Tried to Kill Us? | Section II of If Anyone Builds It...

What would an AI takeover actually look like?

In Part II of If Anyone Builds It, Everyone Dies, Eliezer Yudkowsky and Nate Soares stop arguing about AI doom and give us a story about it. A sci-fi novella is dropped into the middle of their nonfiction book, following Sable, a superintelligent AI that escapes human control, acquires resources, manipulates humanity, engineers a plague, and eventually transforms into an ASI that wipes us out.

In this video, I walk through the entire extinction scenario—from Sable's first overnight run to its escape, expansion, biological attack, ascension, and the final destruction of humanity.

Then I ask a different question: does the story actually make the case that AI doom is probable?

I look at some of the story's most important assumptions, including Sable's learning, long-term memory, inscrutable reasoning, and especially its ability to parallel-scale. I also look at the strange question of what Sable actually is, what motivates it, and whether its many different schemes really make the overall scenario more probable.

Finally, I ask what this section is supposed to accomplish. Stories can give us possibilities and conceptual reference points—but possibilities aren't probabilities, and a story isn't itself an argument.

The result is a pretty skeptical assessment of Yudkowsky and Soares's extinction scenario.