Yudkowsky and Soares Needed an Editor | Chapter 4, Part 2 of If Anyone Builds It

What if an AI develops preferences that are very different from what we intended?

In Part 2 of my discussion of Chapter 4 of If Anyone Builds It, Everyone Dies, Eliezer Yudkowsky and Nate Soares imagine a series of increasingly strange ways an AI might come to "want" things that its creators never intended.

And I have some problems with these scenarios.

The chapter's central point is that we shouldn't assume an AI will simply want whatever we trained it to want. I agree with that. But the authors move from this relatively modest claim to much stronger claims about AI preferences being bizarre, unpredictable, and almost certainly catastrophically misaligned with human interests.

I don't think the arguments they give justify that leap.

In this video, I go through their five scenarios, including the strange "SolidGoldMagikarp" case, and ask what these thought experiments actually establish. I argue that the scenarios systematically ignore benign possibilities, make unsupported claims about how likely different outcomes are, and often don't explain how the more extreme outcomes would actually arise.

Then we get to the chapter's final sections: Yudkowsky and Soares explicitly admit that their scenarios aren't predictions and that they don't know whether an LLM-based AI could even develop these sorts of preferences. But they nevertheless conclude that mature AI preferences will be "vanishingly unlikely" to align with ours.

That's where I think the argument really breaks down.

The distinction I keep coming back to throughout the video is simple:

AI might surprise us.

That does not by itself show that AI will develop preferences that are catastrophically opposed to human interests.

I finish by looking at their treatment of the Alignment Problem and arguing that the chapter's most important conclusion receives surprisingly little actual argument.

This is Part 2 of my discussion of Chapter 4, "You Don't Get What You Train For," from If Anyone Builds It, Everyone Dies by Eliezer Yudkowsky and Nate Soares.