CLIP and DALL·E

On 5 January 2021 OpenAI announced two models. CLIP was trained on 400 million image–caption pairs from the web to do one thing: given a batch of images and a batch of captions, say which goes with which. A text transformer and a vision transformer learned to place matching pairs close together in a shared space. The result classified images it had never been trained on, from written descriptions of the categories, about as well as the best supervised models, and it became the standard way to measure whether a picture matched a sentence.

DALL·E, named for Dalí and the Pixar robot, was a twelve-billion-parameter GPT trained on text and image tokens together, so that it continued a caption with a picture. Its avocado armchairs and radishes walking dogs were the first machine-made images that the public found delightful rather than uncanny.

Between them the two models set up the generative image boom. CLIP scored the candidates and, in later systems, conditioned the generator; diffusion replaced DALL·E's autoregressive decoder within eighteen months. DALL·E 2, Midjourney and Stable Diffusion all use CLIP or a model trained the same way to understand what was asked for.