The Neocognitron

In 1959 Hubel and Wiesel had found that the cat's visual cortex is built from simple cells, which respond to an edge at one position and orientation, and complex cells, which respond to the same edge anywhere in a small region. Kunihiko Fukushima, working at the Japanese broadcaster NHK's research laboratory, turned that finding into a network. The Neocognitron, published in April 1980, alternates layers of S-cells that detect local features with C-cells that pool over position, stacked several times so that later layers respond to larger and more abstract patterns.

The result recognised handwritten digits regardless of where they were drawn, which a perceptron could not do. The architecture is the convolutional network: shared local filters, pooling, and depth. What the Neocognitron lacked was a good way to train it; Fukushima used an unsupervised, layer-by-layer rule.

Yann LeCun supplied the missing piece in 1989 by training the same structure with backpropagation, and AlexNet in 2012 was the Neocognitron with rectified units, dropout and two GPUs. Fukushima published in the depth of the first winter, in a journal of biology, and the field took thirty years to notice how much he had given it.