Long short-term memory

Hochreiter's 1991 thesis had shown why recurrent networks forgot: the gradient vanished as it travelled back through time. The Long Short-Term Memory, published with Jürgen Schmidhuber in Neural Computation on 15 November 1997, is the fix. Each cell holds a value that passes from step to step unchanged unless a learned gate decides to write to it or erase it, so the error signal can flow back through hundreds of steps without shrinking. The network learns not just what to compute but what to remember.

It took a decade to matter, because it was slow to train and the problems it solved, long sequences, were not yet the problems people had data for. Then, from about 2009, Alex Graves and others showed LSTMs beating everything on handwriting and speech, and by 2015 it was inside Google's speech recogniser, Apple's Siri, Amazon's Alexa and, the following year, Google Translate. Sequence-to-sequence learning and the first attention mechanisms were built on it.

The transformer of 2017 replaced the LSTM in language by removing recurrence altogether, which made training parallel. But for the years in which "neural network" came to mean something that could handle text and sound, the LSTM was the network. It is the most cited neural-network paper of the twentieth century.