BERT

GPT read left to right, because it was trained to predict the next word. Jacob Devlin's idea, posted on 11 October 2018, was to hide fifteen percent of the words in a passage at random and train a transformer encoder to guess them from both sides. The model could then use everything around a word to understand it. Pre-trained on Wikipedia and the BooksCorpus, BERT set records on eleven benchmarks at once, including a reading-comprehension test on which it exceeded human performance, and its large version had 340 million parameters.

It was released with code and weights, and within months there were hundreds of variants, RoBERTa, ALBERT, DistilBERT, a whole "BERTology" of papers probing what it had learned. Google put it into Search in October 2019 and said it was the largest improvement in five years. For three years it was the default way to do anything with text that did not involve writing it.

BERT's fill-in-the-blank objective turned out to be the wrong bet for the largest models, because it does not generate, and after GPT-3 the field consolidated on decoders. But the encoders that embed queries, documents and images for search and retrieval are its descendants, and the transformer's takeover of language was completed here.