Dropout

A large network with more parameters than examples will memorise its training data and fail on anything new. The classical remedies penalised large weights or stopped training early. Geoffrey Hinton's proposal, posted to arXiv on 3 July 2012, was stranger: on every training example, delete half the hidden units at random, train the network that remains, and at test time use the full network with the weights halved. Units that could not count on their neighbours being present had to learn features that were useful on their own.

Hinton has said the idea came from thinking about sexual reproduction, which breaks up co-adapted sets of genes, and from a bank teller who kept being rotated to prevent fraud. Whatever its origin, it worked. Error rates fell on every benchmark tried, and the approximate explanation, that training with dropout is like averaging an exponential number of thinned networks, gave it respectability.

AlexNet, trained by two of the paper's authors that autumn, used dropout in its fully connected layers and would have overfit badly without it. The method became standard for a decade. The transformers of the 2020s, trained on data too large to memorise, mostly dropped it again.