Support-vector machines

Vladimir Vapnik had spent the 1970s in Moscow developing, with Alexey Chervonenkis, a theory of when learning from examples can be trusted, and the idea that a classifier should be chosen not merely to fit the data but to fit it with room to spare. At Bell Labs in the early 1990s that became an algorithm. The support-vector machine draws the boundary between two classes that leaves the largest possible margin on either side, and it depends only on the examples nearest the boundary, the support vectors. A trick with kernel functions lets it draw that boundary in an enormous implicit feature space without ever computing the features.

Corinna Cortes and Vapnik's 1995 paper showed it beating everything else on the handwritten-digit benchmark, including the neural networks down the corridor. Through the late 1990s and 2000s the SVM was the default method: it had a theory, a convex optimisation with one answer, and few knobs to tune. Neural networks, by comparison, looked like alchemy.

Deep learning beat it in 2012 on the problems that mattered, and the kernel methods receded to the corners of the field where data is small. But the margin, the idea that a good model is one that does not fit too tightly, is still the language of generalisation.