Adam
Kingma and Ba's optimiser adapts the learning rate for every parameter from running averages of the gradient and its square; it becomes the default way to train almost everything.
what had to happen · 8 events back to 1943
Every event this one built on, transitively, in order. Direct influences are marked.
00 · One neuron · 1
I · Foundations · 5
W1 · The first winter · 1
II · Connection · 1
- 1986Backpropagationdirect
Gradient descent takes steps of a fixed size in the direction the gradient points, and in a network with millions of parameters that size is wrong for most of them: some need large steps, some tiny, and the right size changes as training goes on. Adam, posted on 22 December 2014 by Diederik Kingma and Jimmy Ba, keeps for every parameter a running average of its gradient and of its squared gradient, and divides one by the square root of the other. Parameters with consistently large gradients get smaller steps; parameters with rare, noisy gradients get larger ones. Momentum is built in.
It combined two earlier ideas, AdaGrad and RMSProp, with a bias correction that made it behave well from the first step, and it worked on almost everything with its default settings. That was the point: a practitioner could use it without tuning, and the paper's authors have said the name stands for adaptive moment estimation and not for anyone in particular.
Adam and its weight-decay variant AdamW trained nearly every transformer of the 2020s. The paper has been cited more than 200,000 times, which makes it the most cited paper in machine learning and among the most cited in any science.
what it led to · 0 events downstream
A leaf, for now. Nothing in the archive has built on it yet.