Can Humanity Conquer a Rogue ASI? | Chapter 6

Would humans really have no chance against a superintelligent AI?

In Chapter 6 of If Anyone Builds It, Everyone Dies, Eliezer Yudkowsky and Nate Soares argue that if we end up in conflict with a superintelligent AI, we should expect to lose. Their central analogy is the Aztecs facing the Spanish: the technological gap was so large that the Aztecs couldn't even anticipate the ways their opponents could defeat them.

There's something to this argument. A sufficiently advanced AI might have capabilities that we simply can't foresee.

But I think the Aztec analogy is much less convincing than Yudkowsky and Soares think it is.

So in this video, I look at their argument, dig into what actually happened during the Spanish conquest of the Aztec Empire, and ask whether technological superiority really explains the outcome. The answer is considerably more complicated than "the Spanish were smarter and had better technology."

And that raises a broader philosophical problem with the argument: if we're allowed to imagine an arbitrarily capable AI using methods we can't anticipate, why can't we imagine humans finding some way to stop it?

Would we lose?

Maybe. But I'm not convinced that our ability to imagine hypothetical AI victories tells us very much about what would actually happen.

This is Chapter 6 of my ongoing series on If Anyone Builds It, Everyone Dies by Eliezer Yudkowsky and Nate Soares.