Scaling laws for neural language models

The paper posted on 23 January 2020 asked a plain empirical question: if you train transformers of different sizes on different amounts of text with different budgets of compute, how does the loss behave? The answer was a set of straight lines on log-log axes. Test loss fell as a power law in the number of parameters, in the size of the dataset and in the compute spent, each across many orders of magnitude, with no sign of levelling off. Architecture details mattered little; the exponents were what they were; and a model's performance could be predicted before it was trained.

That made scale a plan rather than a hope. Jared Kaplan's team, several of whom left to found Anthropic the next year, showed that for a fixed compute budget the best results came from a large model trained on relatively little data, stopped early, which is the recipe GPT-3 followed four months later. DeepMind's Chinchilla paper of 2022 corrected the exponents and found the data had been undervalued, but the shape of the result held.

The scaling laws are the reason the labs spent billions on compute, the reason the models kept improving, and the instrument on this page. Drag the compute and the loss follows the line.