Moore's law

The 19 April 1965 issue of Electronics carried a four-page article by Gordon Moore, then director of research at Fairchild Semiconductor, with a graph of five data points. The number of components that could be put on an integrated circuit at minimum cost had doubled every year since 1959, and he expected the trend to continue for at least ten. In 1975 he revised the period to two years, and the industry organised itself around hitting the target.

Nothing in the article is about intelligence. It is on this timeline because it is the reason the ideas of the 1940s eventually worked. The perceptron of 1958 ran on an IBM 704 that could do about 12,000 additions a second. AlexNet in 2012 trained on two GPUs doing about 10¹² operations a second, and GPT-4 in 2023 on tens of thousands of chips for months. The algorithms had improved; the hardware had improved by a factor of a hundred million or more.

Richard Sutton's "bitter lesson" of 2019 is Moore's law read as a research strategy: the methods that win are the ones that scale with compute, and the hand-built cleverness that beats them today loses in a few doublings.