Sequence to sequence learning

Machine translation in 2014 was a pipeline of statistical components, alignment, phrase tables, reordering, language models, engineered over twenty years. Ilya Sutskever, Oriol Vinyals and Quoc Le at Google replaced it with one network. An LSTM read the English sentence and produced a vector; a second LSTM read the vector and wrote the French, one word at a time. Trained on twelve million sentence pairs, four layers deep, with a trick of reversing the input so the beginnings of both sentences were close together, it matched the best phrase-based system on the WMT benchmark and beat it when the two were combined.

Kyunghyun Cho's group had published the same architecture in June; Bahdanau's attention had appeared nine days before Sutskever's preprint. Together the three papers established neural machine translation, and Google replaced its production system with one in 2016.

The larger point was the shape. A sequence goes in, a sequence comes out, and the network learns the mapping from examples: translation, summarisation, question answering, code generation and conversation are all the same problem. Every language model since is a sequence-to-sequence system, and Sutskever's insistence that scale would keep improving it took him to OpenAI as chief scientist the following year.