An image is worth 16×16 words

Convolutional networks had owned vision since 2012 because their design encoded what everyone knew about images: nearby pixels matter, features repeat across positions. The vision transformer, posted on 22 October 2020, encoded none of it. It sliced an image into a grid of 16-by-16-pixel patches, treated each as a word, and fed the sequence to a plain transformer encoder. On ImageNet alone it lost to convolutional networks. Pre-trained on 300 million images from Google's private dataset, it beat them.

The result was the bitter lesson applied to sight. The inductive biases that had made convolution the right architecture for small data became a handicap at large scale, where the model could learn locality for itself, and the same transformer that read text now read pictures.

That convergence shaped what came next. CLIP, three months later, trained a text transformer and an image transformer to agree, which made text-to-image generation possible; the multimodal models of the 2020s take words, pixels, audio and video as one stream of tokens because the vision transformer showed they could. Convolutions did not disappear, but the era in which each kind of data had its own architecture ended here.