Google Translate goes neural

Two years after the sequence-to-sequence and attention papers, Google put them into production. The system described on 26 September 2016 was an eight-layer LSTM encoder and decoder with attention between them, trained on Google's translation data and run on TPUs, and it cut translation errors by 55 to 85 percent on the major language pairs in human evaluations. It went live for Chinese-to-English that day and for all of Translate's languages within a year.

The paper is a catalogue of the engineering needed to make research work at scale: wordpiece tokenisation to handle rare words, which later became the tokeniser of BERT; residual connections between layers; quantised inference to make it fast enough; and a trick of training one model on many languages at once that allowed translation between pairs it had never seen together.

The switch is on this timeline because it was the first time a deep network replaced a whole classical pipeline in a product used by hundreds of millions of people, and because it happened eight months before the transformer made the LSTM obsolete. The translation problem, sixty-two years after the Georgetown demonstration and fifty after ALPAC, had been largely solved, and the solution was thrown away within a year for a better one.