Residual networks

Deeper networks should be at least as good as shallower ones, since the extra layers could simply pass their input through. In practice, by 2015, networks past about twenty layers got worse, not from overfitting but because the optimiser could not find the pass-through solution. Kaiming He's group at Microsoft Research Asia made the pass-through the default. Each block of layers computes a change to its input and adds it, so a block that learns nothing does nothing, and the gradient has a clear path back through the identity connections however deep the stack.

The paper, posted on 10 December 2015, trained a 152-layer network on ImageNet and won the 2015 challenge with 3.57 percent top-five error, below the estimated human rate, along with the detection and segmentation competitions. A 1,000-layer version trained without difficulty.

Residual connections are now in everything. The transformer of 2017 is a stack of residual blocks, and the reason language models can be a hundred layers deep is the identity path He introduced for pictures. The paper is among the most cited in computer science, and its central idea, that a layer should learn a small correction rather than a whole new representation, is the last piece of the deep-learning architecture that was needed.