GPT-3

GPT-3 was the scaling laws carried out. Posted on 28 May 2020, it had 175 billion parameters, a hundred times GPT-2, and was trained on about 300 billion tokens of filtered web text, books and Wikipedia at a cost estimated at several million dollars. Its size was the headline. The finding that mattered was in the title: it learned tasks from a few examples written into the prompt. Show it three English-to-French pairs and it translated the fourth; show it a format and it filled it in. Nothing in the model changed. The abilities were already inside it, and the prompt selected them.

In-context learning was the surprise the scaling laws had not predicted, and it changed how models were used. Instead of collecting a dataset and fine-tuning, one wrote instructions in English. The paper also reported that people could not reliably tell its news articles from human ones, and devoted a section to the risks.

OpenAI did not release the weights. It sold access through an API from June 2020, the first time a frontier model was a product, and licensed the model exclusively to Microsoft that September. The company founded to keep AI open had become the company that kept its models closed, and it had built the thing that, refined for two years, would be ChatGPT.