Deep belief networks and the word 'deep'

By 2006 Geoffrey Hinton had spent two decades on neural networks through two winters, funded latterly by a Canadian institute that had decided to back unfashionable ideas. That summer two papers came out of his Toronto group. The first, in Neural Computation, showed how to train a deep network by treating each pair of layers as a restricted Boltzmann machine, training them greedily from the bottom up, and then fine-tuning the whole stack with backpropagation. The second, in Science, used the method to build deep autoencoders that compressed images and documents far better than the standard techniques.

The trick got around the vanishing gradient by not relying on it: the layers were already sensible before backpropagation started. Deep networks, which had been unstable and unfashionable, became trainable, and the phrase "deep learning" was adopted to describe them, partly as a rebrand for a field that had learned to avoid the words neural network.

The layer-wise pretraining was abandoned within a few years, once rectified units, better initialisation and GPUs made it unnecessary. What survived was the confidence. The 2006 papers convinced a small group of people that depth was the direction, and that group produced the results of 2012.