Genetic algorithms

John Holland had been thinking about evolution as a search procedure since the 1960s, and his 1975 book made it a method. Represent candidate solutions as strings, keep a population of them, score each one, breed the better ones by cutting and splicing their strings, mutate a little, and repeat. Holland's schema theorem gave a mathematical argument for why the procedure tends to find good solutions, and his students at Michigan turned it into a field.

Genetic algorithms need no gradient, no differentiable model and no understanding of the problem beyond a score, and through the winters of the 1970s and 80s they kept a form of learning alive in engineering departments where neural networks were out of fashion. They designed antennas for NASA, schedules for factories and circuits that human engineers did not understand.

In the deep-learning era they returned as neural architecture search and as evolution strategies for reinforcement learning, where computing a gradient is expensive and running many copies is cheap. Holland's population, selection and variation are also the frame in which the 2020s debate about AI systems improving other AI systems is usually posed.