Google Brain's network discovers cats

Google Brain began in 2011 as a collaboration between Andrew Ng, Jeff Dean and Greg Corrado to find out what happened when the deep networks of Ng's Stanford group were trained on Google's computers. The answer, presented at ICML in June 2012 and on the front page of the New York Times, was a network with a billion connections, trained for three days on 16,000 processor cores using frames from ten million YouTube videos, with no labels at all. Among its top-level units was one that responded to cat faces, and another to human faces, which nobody had asked for.

The result was modest as vision, the network's accuracy on ImageNet categories was 15.8 percent, and enormous as a demonstration. It showed that a large enough network with enough data would organise its own concepts, and that the limiting factor was compute. Within months Google had built the infrastructure, DistBelief, that became TensorFlow, and had hired Geoffrey Hinton.

The paper also fixed a number in the public mind. A billion parameters, in 2012, was a headline; GPT-3 would have 175 billion eight years later and the frontier models of the 2020s more than a trillion. The cat neuron was the first widely reported evidence that the way to make networks smarter was to make them bigger.