Attention is all you need

The paper posted on 12 June 2017 was a translation paper. Its eight authors, listed in an order they said was random, had found that the recurrent networks everyone used for sequences were unnecessary. Bahdanau's attention, which let a decoder look back over the source, could be turned inward: every position in a sentence attends to every other position, computing what to weigh from learned queries and keys, and the whole layer runs at once rather than word by word. Stack six of those layers, with residual connections and normalisation between them, add a position signal so the model knows word order, and you have the transformer. It beat the best translation systems on English–German and English–French and trained in a fraction of the time.

Speed was the point. A recurrent network processes a thousand-word document in a thousand sequential steps; a transformer processes it in one, which means it can use every core of a GPU pod, which means it can be trained on far more data. Scale, which the field had been suspecting was the answer since 2012, became possible for language.

Every model on this timeline after 2017 that reads or writes is a transformer: BERT, the GPTs, AlphaFold 2, Stable Diffusion's text encoder, the vision transformer, the models that drive cars and fold proteins and write code. The paper has been cited more than 150,000 times. The instrument below runs one on the sentence you type and shows you the attention.