Reverse-mode automatic differentiation

In 1970 a Finnish master's student, Seppo Linnainmaa, was studying how rounding errors accumulate through a long computation. To track them he needed the derivative of the final result with respect to every intermediate value, and he found that all of them could be obtained by a single pass backwards through the computation, applying the chain rule in reverse. The cost was about the same as running the computation forwards once, however many inputs there were.

He was not thinking about neural networks and the thesis, in Finnish, was not read by anyone who was. But the reverse mode of automatic differentiation is exactly the algorithm that a network needs to learn: propagate the error backwards from the output, and each weight receives the derivative of the loss with respect to itself. Paul Werbos applied the idea to networks in 1974. Rumelhart, Hinton and Williams made it famous in 1986.

Every deep learning framework, from Theano to PyTorch, is at bottom an implementation of Linnainmaa's backward sweep over an arbitrary graph of operations. The word autograd is his idea with a shorter name. He received little credit for decades; Andreas Griewank's 2012 history of the question is titled with it.