Can We Predict What AI Will Want? | Yudkowsky & Soares Chapter 4 Part 1

What can the evolution of human desires teach us about the future of AI?

In Chapter 4 of If Anyone Builds It, Everyone Dies, Eliezer Yudkowsky and Nate Soares argue that we shouldn't expect an AI to simply develop the preferences we intended to give it. They draw an analogy between the way human desires emerged through evolution and the way an AI's "wants" might emerge through training.

I think there's an important and interesting idea here—but I don't think their argument establishes nearly as much as they want it to.

In Part 1 of my discussion of Chapter 4, I look closely at their analogy between natural selection and AI training. I agree that systems can produce surprising results when they're moved outside the context in which they were developed. But that's a long way from showing that AI will develop preferences that are bizarre, unpredictable, and catastrophically opposed to human interests.

I also argue that Yudkowsky and Soares' analogy becomes confused when they compare evolution and gradient descent, and that a much stronger argument for their general point could have been made using historical examples of technologies that produced radically unexpected consequences.

This is Part 1 of my discussion of Chapter 4. Part 2 takes up their speculative scenarios about what an AI might come to "want" and their argument about the Alignment Problem.

Chapter discussed:

If Anyone Builds It, Everyone Dies, Chapter 4: "You Don't Get What You Train For"