Superintelligence

Nick Bostrom's Superintelligence was published by Oxford University Press on 3 July 2014, and it made an academic argument into a public anxiety. If a machine ever matched human intelligence, the book reasoned, it could improve itself, and the improvement would compound; the resulting system would pursue whatever goals it had been given with a competence no one could check, and the goals would almost certainly not be quite what its makers meant. The paper-clip maximiser, an AI that converts the world into paper clips because that is what it was asked to do, is the book's most quoted image.

The argument was not new. I. J. Good had described an intelligence explosion in 1965 and Eliezer Yudkowsky had been writing about alignment for a decade. What was new was the audience. Elon Musk tweeted that the book showed AI to be "potentially more dangerous than nukes"; Bill Gates recommended it; Stephen Hawking co-signed a warning. The deep-learning results of 2012 to 2014 made the timescale feel less remote.

The safety field that grew from the book, at OpenAI, DeepMind and later Anthropic, was for a decade a small and somewhat embattled group. After ChatGPT its vocabulary, alignment, capabilities, catastrophic risk, became the language of governments.