The Case for Superintelligence Is Surprisingly Thin | If Anyone Builds It... Chapter 1

What makes humans so powerful? According to Eliezer Yudkowsky and Nate Soares in If Anyone Builds It, Everyone Dies, the answer is intelligence—and intelligence is fundamentally about predicting the world and steering it.

In this episode of Ontology Explained, I take a critical look at the book's opening argument.

I agree with the basic idea that human intelligence has played a major role in our extraordinary impact on the planet. But I think the book moves far too quickly over some very difficult questions. What actually makes humans special? What is intelligence? Is prediction really one half of intelligence and steering the other? And why should we be so confident that AI will inevitably surpass human intelligence?

I'll look at the book's brief story about human dominance, its conception of intelligence as prediction and steering, the philosophical alternative of beliefs and desires, and the arguments for why AI will eventually eclipse humanity.

My main concern is not that the book's conclusions are obviously wrong. It's that some of its most important premises are surprisingly underdeveloped. There is little engagement with anthropology, philosophy of mind, or the existing philosophical literature on concepts like belief, desire, and direction of fit.

And that's a problem, because the possibility of superintelligent AI is doing a lot of work in the argument. If ASI is genuinely plausible, then its potential consequences are worth taking very seriously. But we need a stronger argument for why we should believe it's coming.

This is my ongoing series reviewing and critiquing If Anyone Builds It, Everyone Dies. I'll continue on to Chapter 2—but with some skepticism about the foundations laid in Chapter 1.