GPT: generative pre-training

The recipe that produced every later GPT is in a paper OpenAI posted on 11 June 2018 with little fanfare. Take the decoder half of the transformer, twelve layers and 117 million parameters. Train it on the BooksCorpus, 7,000 unpublished novels, to predict each word from the ones before it, a task that needs no labels and for which the world's text is the dataset. Then fine-tune the same network briefly on each downstream task, question answering, textual entailment, sentence similarity, and it beats the specialised models on nine of twelve benchmarks.

The result confirmed what Elman had seen in miniature in 1990: predicting the next word forces a model to learn grammar, facts and reasoning as by-products. Alec Radford's paper made pre-training on raw text the foundation of natural-language processing, and it made the objective, next-token prediction, the one that would be scaled for the next eight years.

Google's BERT, four months later, used the encoder half and a fill-in-the-blank objective and was for a while the more influential of the two. But BERT could not generate text, and GPT could, and generation is what the public eventually got.