Will Superintelligent AI Want to Kill Us? | Chapter 5

In Chapter 5 of If Anyone Builds It, Everyone Dies, Eliezer Yudkowsky and Nate Soares make their clearest case yet for why a superintelligent AI would want humanity gone.

Their central argument is built around a strange thought experiment: alien birds who desperately care about building "correct nests"—nests containing a prime number of stones. The idea is that just as we can't imagine why an alien intelligence would care about our values, an ASI won't care about ours.

But does the analogy actually work?

In this video, I reconstruct the argument of Chapter 5 and examine some of its key assumptions: moral relativism, the idea that an ASI's values would necessarily conflict with ours, instrumental convergence, the "billionaire argument," and the assumption that a sufficiently powerful AI would inevitably dominate and eliminate us.

I also look at one of the chapter's stranger turns: the suggestion that a future without humanity wouldn't be worth having because machines wouldn't create a world filled with "wonder and joy and humor and love."

By the end, I think Chapter 5 contains the book's most coherent argument so far—but I'm still not convinced they've met the burden of proof.