Werbos applies backpropagation to neural networks

Paul Werbos was a Harvard doctoral student in applied mathematics who wanted to forecast political behaviour, and in 1974 he submitted a thesis titled Beyond Regression: New Tools for Prediction and Analysis in the Behavioral Sciences. Buried in it is a general method for computing how the output of a layered system changes with each of its parameters, by passing derivatives backwards through the layers, and an explicit application to training the multi-layer perceptrons that Minsky and Papert had dismissed five years earlier.

He had tried to publish the idea in 1971 and 1972 and been discouraged, in part by Minsky himself. The thesis was accepted, shelved, and cited by almost no one for a decade. The neural-network community rediscovered the method in the early 1980s, and Werbos's priority was acknowledged only after the 1986 paper made it famous.

The episode is a lesson about winters. The key algorithm of the deep-learning era existed in full, in an American university library, two years before the first winter was declared, and sat there through it. What was missing was not the idea but the willingness to fund anyone to try it, and computers fast enough for the trial to be convincing.