Backpropagation

The paper is three pages long and appeared in Nature on 9 October 1986. Its claim is in the title: a network with hidden layers can learn representations, and the way to train it is to compute the error at the output and pass it backwards, layer by layer, using the chain rule, so that every weight in the network receives its share of the blame. The gradient that results is the direction to move, and moving a little at a time is gradient descent. David Rumelhart had worked it out in 1982; Geoffrey Hinton and Ronald Williams helped him make it convincing.

The mathematics was not new. Linnainmaa had the algorithm in 1970 and Werbos had applied it to networks in 1974. What the 1986 paper did was demonstrate, on problems people cared about, that the hidden units learned something interpretable, that the network solved XOR and family-tree relationships and the encoder problems Minsky and Papert had used as evidence of hopelessness. It arrived in the two-volume Parallel Distributed Processing books the same year, and a generation of researchers learned it from there.

Everything in the deep-learning era is trained this way. Convolutional networks, LSTMs, transformers, diffusion models: the architectures differ, and the loss functions differ, but the weights are always set by backpropagation and some variant of gradient descent. Rumelhart died in 2011. Hinton's 2024 Nobel citation begins with this paper.