# Chain-of-thought prompting

Date: 28 January 2022 · Era: VI · Everyone · Category: theory · Significance: 3/5
People: Jason Wei, Denny Zhou, Quoc Le
Organisations: Google Research, Brain Team

Canonical: https://shapeofintelligence.com/timeline/2022-chain-of-thought/ · Dataset and text CC BY 4.0

> Wei and colleagues at Google show that asking a large model to write out its reasoning steps before answering roughly triples its accuracy on maths problems; thinking out loud becomes a technique.

## What had to happen

36 events back to 1943, oldest first. Direct influences are marked *direct*.

- [1943 · A logical calculus of nervous activity](https://shapeofintelligence.com/md/timeline/1943-mcculloch-pitts-neuron/)
- [1949 · Cells that fire together wire together](https://shapeofintelligence.com/md/timeline/1949-hebb-organization-of-behavior/)
- [1958 · The perceptron learns](https://shapeofintelligence.com/md/timeline/1958-perceptron/)
- [1960 · ADALINE and the least-mean-squares rule](https://shapeofintelligence.com/md/timeline/1960-adaline-lms/)
- [1965 · Moore's law](https://shapeofintelligence.com/md/timeline/1965-moores-law/)
- [1969 · Perceptrons](https://shapeofintelligence.com/md/timeline/1969-perceptrons-book/)
- [1970 · Reverse-mode automatic differentiation](https://shapeofintelligence.com/md/timeline/1970-linnainmaa-backprop/)
- [1974 · Werbos applies backpropagation to neural networks](https://shapeofintelligence.com/md/timeline/1974-werbos-backprop/)
- [1980 · The Neocognitron](https://shapeofintelligence.com/md/timeline/1980-neocognitron/)
- [1982 · The Hopfield network](https://shapeofintelligence.com/md/timeline/1982-hopfield-network/)
- [1985 · The Boltzmann machine](https://shapeofintelligence.com/md/timeline/1985-boltzmann-machine/)
- [1986 · Backpropagation](https://shapeofintelligence.com/md/timeline/1986-backpropagation/)
- [1989 · LeNet reads handwritten postcodes](https://shapeofintelligence.com/md/timeline/1989-lenet/)
- [1990 · Finding structure in time](https://shapeofintelligence.com/md/timeline/1990-elman-network/)
- [1991 · The vanishing gradient problem](https://shapeofintelligence.com/md/timeline/1991-vanishing-gradient/)
- [1997 · Long short-term memory](https://shapeofintelligence.com/md/timeline/1997-lstm/)
- [1998 · MNIST and LeNet-5](https://shapeofintelligence.com/md/timeline/1998-mnist-lenet5/)
- [1999 · The first GPU](https://shapeofintelligence.com/md/timeline/1999-geforce-256/)
- [2003 · A neural probabilistic language model](https://shapeofintelligence.com/md/timeline/2003-neural-language-model/)
- [2006 · Deep belief networks and the word 'deep'](https://shapeofintelligence.com/md/timeline/2006-deep-belief-nets/)
- [2007 · CUDA](https://shapeofintelligence.com/md/timeline/2007-cuda/)
- [2009 · Deep learning moves to GPUs](https://shapeofintelligence.com/md/timeline/2009-gpu-deep-learning/)
- [2009 · ImageNet](https://shapeofintelligence.com/md/timeline/2009-imagenet/)
- [2010 · Rectified linear units](https://shapeofintelligence.com/md/timeline/2010-relu/)
- [2012 · Google Brain's network discovers cats](https://shapeofintelligence.com/md/timeline/2012-google-brain-cat/)
- [2012 · Dropout](https://shapeofintelligence.com/md/timeline/2012-dropout/)
- [2012 · AlexNet wins ImageNet](https://shapeofintelligence.com/md/timeline/2012-alexnet/)
- [2014 · Attention](https://shapeofintelligence.com/md/timeline/2014-bahdanau-attention/)
- [2014 · Sequence to sequence learning](https://shapeofintelligence.com/md/timeline/2014-seq2seq/)
- [2015 · Batch normalisation](https://shapeofintelligence.com/md/timeline/2015-batchnorm/)
- [2015 · Residual networks](https://shapeofintelligence.com/md/timeline/2015-resnet/)
- [2017 · Attention is all you need](https://shapeofintelligence.com/md/timeline/2017-attention-is-all-you-need/)
- [2018 · GPT: generative pre-training](https://shapeofintelligence.com/md/timeline/2018-gpt-1/)
- [2019 · GPT-2 and the model too dangerous to release](https://shapeofintelligence.com/md/timeline/2019-gpt-2/)
- [2020 · Scaling laws for neural language models](https://shapeofintelligence.com/md/timeline/2020-scaling-laws/)
- [2020 · GPT-3](https://shapeofintelligence.com/md/timeline/2020-gpt-3/) *direct*

Large language models in 2021 could write fluently and failed at arithmetic word problems a child could do, and the standard explanation was that they did not reason. Jason Wei's paper, posted on 28 January 2022, suggested the failure was in the asking. Instead of showing the model a few problems with answers, show it a few problems with worked solutions, the intermediate steps written in words. The model then wrote steps of its own before answering, and on a benchmark of grade-school maths problems the 540-billion-parameter PaLM went from 18 percent correct to 57. Smaller models did not benefit; the ability appeared only above a certain scale.

The finding was simple enough to be adopted everywhere within months. "Let's think step by step", a five-word prompt that a Tokyo group showed worked on its own, became the most reproduced result of the year. Chain-of-thought was also an early example of an emergent ability, something a large model could do that a smaller one of the same design could not, and it fed the argument about what scale was buying.

It is the ancestor of the reasoning models of 2024, o1 and its successors, which are trained to generate long chains of thought and to search over them, rather than merely prompted to.

## What it led to

11 events downstream, through 2026.

Built on it directly:

- [2022 · PaLM](https://shapeofintelligence.com/md/timeline/2022-palm/)
- [2024 · o1 and reasoning models](https://shapeofintelligence.com/md/timeline/2024-o1/)
- [2025 · Gold at the Mathematical Olympiad](https://shapeofintelligence.com/md/timeline/2025-imo-gold/)

And, through them, by era:

- VI · Everyone: 1 events
- VII · Agents: 7 events

## Sources

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models (arXiv:2201.11903)](https://arxiv.org/abs/2201.11903) — paper

---

The Shape of Intelligence by Jamie McKaye (https://jamiemckaye.com) · Markdown mirror of https://shapeofintelligence.com/timeline/2022-chain-of-thought/

Before: [2022 · InstructGPT](https://shapeofintelligence.com/md/timeline/2022-instructgpt/) · After: [2022 · Chinchilla: the models were undertrained](https://shapeofintelligence.com/md/timeline/2022-chinchilla/)
