The vanishing gradient problem

Backpropagation worked for networks with one or two hidden layers and stalled for deeper ones, and for recurrent networks it could not learn dependencies more than a few steps apart. Nobody knew exactly why until Sepp Hochreiter's 1991 diploma thesis, written under Jürgen Schmidhuber in Munich. The gradient that reaches an early layer is a product of the derivatives of every layer after it, and with sigmoid units those derivatives are less than one. Multiplied together across many layers or time steps, the signal shrinks towards zero; with other weight settings it explodes. Either way the early layers receive nothing useful to learn from.

The thesis was in German and read by few, but the analysis explained a decade of failures and set the agenda for fixing them. Hochreiter and Schmidhuber's LSTM of 1997 built a memory cell whose gradient could pass through unchanged. Rectified linear units, careful initialisation, batch normalisation and residual connections, all of them the standard equipment of the deep-learning era, are answers to the problem stated here.

The thesis is also why "deep" was not a compliment for another fifteen years: until the fixes arrived, depth was the thing that stopped networks learning.