# PaLM

Date: 4 April 2022 · Era: VI · Everyone · Category: model · Significance: 3/5
People: Aakanksha Chowdhery, Sharan Narang, Jacob Devlin
Organisations: Google Research

Canonical: https://shapeofintelligence.com/timeline/2022-palm/ · Dataset and text CC BY 4.0

> Google trains a 540-billion-parameter model across two TPU pods and reports emergent abilities that appear only at scale, explaining jokes and reasoning through problems.

## What had to happen

38 events back to 1943, oldest first. Direct influences are marked *direct*.

- [1943 · A logical calculus of nervous activity](https://shapeofintelligence.com/md/timeline/1943-mcculloch-pitts-neuron/)
- [1949 · Cells that fire together wire together](https://shapeofintelligence.com/md/timeline/1949-hebb-organization-of-behavior/)
- [1958 · The perceptron learns](https://shapeofintelligence.com/md/timeline/1958-perceptron/)
- [1960 · ADALINE and the least-mean-squares rule](https://shapeofintelligence.com/md/timeline/1960-adaline-lms/)
- [1965 · Moore's law](https://shapeofintelligence.com/md/timeline/1965-moores-law/)
- [1969 · Perceptrons](https://shapeofintelligence.com/md/timeline/1969-perceptrons-book/)
- [1970 · Reverse-mode automatic differentiation](https://shapeofintelligence.com/md/timeline/1970-linnainmaa-backprop/)
- [1974 · Werbos applies backpropagation to neural networks](https://shapeofintelligence.com/md/timeline/1974-werbos-backprop/)
- [1980 · The Neocognitron](https://shapeofintelligence.com/md/timeline/1980-neocognitron/)
- [1982 · The Hopfield network](https://shapeofintelligence.com/md/timeline/1982-hopfield-network/)
- [1985 · The Boltzmann machine](https://shapeofintelligence.com/md/timeline/1985-boltzmann-machine/)
- [1986 · Backpropagation](https://shapeofintelligence.com/md/timeline/1986-backpropagation/)
- [1989 · LeNet reads handwritten postcodes](https://shapeofintelligence.com/md/timeline/1989-lenet/)
- [1990 · Finding structure in time](https://shapeofintelligence.com/md/timeline/1990-elman-network/)
- [1991 · The vanishing gradient problem](https://shapeofintelligence.com/md/timeline/1991-vanishing-gradient/)
- [1997 · Long short-term memory](https://shapeofintelligence.com/md/timeline/1997-lstm/)
- [1998 · MNIST and LeNet-5](https://shapeofintelligence.com/md/timeline/1998-mnist-lenet5/)
- [1999 · The first GPU](https://shapeofintelligence.com/md/timeline/1999-geforce-256/)
- [2003 · A neural probabilistic language model](https://shapeofintelligence.com/md/timeline/2003-neural-language-model/)
- [2006 · Deep belief networks and the word 'deep'](https://shapeofintelligence.com/md/timeline/2006-deep-belief-nets/)
- [2007 · CUDA](https://shapeofintelligence.com/md/timeline/2007-cuda/)
- [2009 · Deep learning moves to GPUs](https://shapeofintelligence.com/md/timeline/2009-gpu-deep-learning/)
- [2009 · ImageNet](https://shapeofintelligence.com/md/timeline/2009-imagenet/)
- [2010 · Rectified linear units](https://shapeofintelligence.com/md/timeline/2010-relu/)
- [2012 · Google Brain's network discovers cats](https://shapeofintelligence.com/md/timeline/2012-google-brain-cat/)
- [2012 · Dropout](https://shapeofintelligence.com/md/timeline/2012-dropout/)
- [2012 · AlexNet wins ImageNet](https://shapeofintelligence.com/md/timeline/2012-alexnet/)
- [2014 · Attention](https://shapeofintelligence.com/md/timeline/2014-bahdanau-attention/)
- [2014 · Sequence to sequence learning](https://shapeofintelligence.com/md/timeline/2014-seq2seq/)
- [2015 · Batch normalisation](https://shapeofintelligence.com/md/timeline/2015-batchnorm/)
- [2015 · Residual networks](https://shapeofintelligence.com/md/timeline/2015-resnet/)
- [2016 · Google reveals the TPU](https://shapeofintelligence.com/md/timeline/2016-tpu/) *direct*
- [2017 · Attention is all you need](https://shapeofintelligence.com/md/timeline/2017-attention-is-all-you-need/)
- [2018 · GPT: generative pre-training](https://shapeofintelligence.com/md/timeline/2018-gpt-1/)
- [2019 · GPT-2 and the model too dangerous to release](https://shapeofintelligence.com/md/timeline/2019-gpt-2/)
- [2020 · Scaling laws for neural language models](https://shapeofintelligence.com/md/timeline/2020-scaling-laws/)
- [2020 · GPT-3](https://shapeofintelligence.com/md/timeline/2020-gpt-3/) *direct*
- [2022 · Chain-of-thought prompting](https://shapeofintelligence.com/md/timeline/2022-chain-of-thought/) *direct*

The Pathways Language Model, announced on 4 April 2022, was the largest dense transformer disclosed at the time, 540 billion parameters trained on 780 billion tokens across 6,144 TPU v4 chips in two pods, using a new system, Pathways, to spread one model across data centres. Its paper is 87 pages, and much of it is a catalogue of things the model could do that its smaller versions could not: explain why a joke was funny, follow chains of reasoning through multi-step problems, and, with chain-of-thought prompting, match fine-tuned models on maths.

The paper made "emergent abilities" a term of art, with a graph of tasks whose accuracy sat at chance until a certain scale and then jumped. Whether the jumps were real or artefacts of how the tasks were scored became a debate that ran for two years. Either way, PaLM was Google's demonstration that it could match OpenAI's scale, eight months before ChatGPT made the question commercial.

PaLM 2 in May 2023 ran Bard; Gemini replaced both in December. The model is on this timeline as the high point of the pure-scale era, before Chinchilla's data correction and human feedback changed what the laboratories optimised.

## What it led to

2 events downstream, through 2025.

Built on it directly:

- [2023 · Gemini](https://shapeofintelligence.com/md/timeline/2023-gemini/)

And, through them, by era:

- VII · Agents: 1 events

## Sources

- [PaLM: Scaling Language Modeling with Pathways (arXiv:2204.02311)](https://arxiv.org/abs/2204.02311) — paper

---

The Shape of Intelligence by Jamie McKaye (https://jamiemckaye.com) · Markdown mirror of https://shapeofintelligence.com/timeline/2022-palm/

Before: [2022 · Chinchilla: the models were undertrained](https://shapeofintelligence.com/md/timeline/2022-chinchilla/) · After: [2022 · DALL·E 2](https://shapeofintelligence.com/md/timeline/2022-dalle-2/)
