VI · Attention, transformers and pretraining, from 2017
The context
How did attention and transformers change the picture?
In 2017 eight researchers at Google described the transformer, a network that reads a sentence all at once and lets every word attend to every other. Attention weights say which words matter for which. Training is simple to state: predict the next token, again and again, on more text than a person could read in a thousand lifetimes. What the model learns is a representation shaped by context: the same word means different things after different words, and the model's numbers move accordingly. By 2018 BERT set records across language tasks; in 2019 GPT-2 wrote paragraphs; in 2020 GPT-3, with 175 billion parameters, did tasks it had never been trained for when shown a few examples in its prompt. Prediction turned out to produce behaviour. Whether the model understands anything is a question people still argue about. What it does is measurable: a distribution over what comes next.
One word. Different worlds.
Try the context“River” gives “bank” a geographical context: the land beside water.
An authored illustration of contextual meaning. The lines are explanatory connections, not attention weights measured from a model; words are shown whole rather than as model tokens.
Artefacts
Attention Is All You Need
12 June 2017 · reconstruction
The first page of "Attention Is All You Need" (Vaswani and others, 2017), set again in the exhibition's type. A reconstruction, not the arXiv page. Reconstruction; the paper is Vaswani and others, 2017. reconstructed · Reconstruction; the paper is Vaswani and others, 2017 · n/a (made for the exhibition) · source
From the archive
Claims in this chapter
GPT-3 has 175 billion parameters.
unverified · Language Models are Few-Shot Learners (arXiv:2005.14165)