Chinchilla: the models were undertrained

Kaplan's 2020 scaling laws had implied that, for a fixed budget of compute, one should build a very large model and train it on relatively little data, and the industry had done so: GPT-3, Gopher and Megatron-Turing had between 175 and 530 billion parameters and had each seen about 300 billion tokens. Jordan Hoffmann's paper, posted on 29 March 2022, trained over 400 models to redo the measurement and found the earlier fit had been distorted by its learning-rate schedule. For compute-optimal training, parameters and training tokens should scale in equal proportion, about twenty tokens per parameter.

To prove it they trained Chinchilla, 70 billion parameters on 1.4 trillion tokens, the same compute as Gopher's 280 billion parameters on 300 billion tokens, and it beat Gopher, GPT-3 and the rest on almost every benchmark. The existing giants had been, in the paper's word, undertrained.

The result reset the field's recipe. Meta's LLaMA the following year trained small models far past the Chinchilla point because inference cost, not training cost, was what mattered for a deployed model, and the models of the mid-2020s train on tens of trillions of tokens. The data, not the parameters, became the constraint, and the search for more of it, and for synthetic substitutes, followed.