Batch normalisation

As a deep network trains, each layer's inputs shift because the layers below it are changing, and every layer is chasing a moving target. Sergey Ioffe and Christian Szegedy's fix, posted on 11 February 2015, was to standardise each layer's inputs over the current mini-batch, subtracting the mean and dividing by the standard deviation, and then to let the network learn a scale and shift of its own. The paper's explanation, reducing internal covariate shift, was later disputed; the effect was not. Networks trained with batch normalisation reached the same accuracy in fourteen times fewer steps, tolerated much higher learning rates, and needed less dropout.

The immediate result was that Google's Inception network beat human-level accuracy on ImageNet, at 4.8 percent top-five error, in the same paper. The longer result was depth. ResNet later that year stacked 152 layers, and could not have been trained without normalisation inside it.

The technique's dependence on the batch made it awkward for sequences, and the transformer of 2017 used layer normalisation, which standardises across a layer's units for each example instead. But the principle, keep the activations well scaled and the gradients will behave, is one of the reasons networks of the 2020s can be a thousand layers deep.