The universal approximation theorem

Perceptrons had shown what a single layer could not represent. In 1989 two groups showed what two layers could: everything, near enough. George Cybenko proved that a network with one hidden layer of sigmoid units and a linear output can approximate any continuous function on a bounded domain as closely as you like, given enough units. Kurt Hornik, Maxwell Stinchcombe and Halbert White proved the same result more generally the same year, for a broad class of activation functions.

The theorem settled the representational argument that had hung over the field for twenty years and moved the difficulty somewhere else. A network that can represent anything is only useful if you can find the weights, and if the number of units required is not absurd. The 1990s discovered that both were hard: training deep networks was unstable and shallow ones needed exponentially many units for some functions.

The practical answer, when it came, was depth. Deep networks represent compositional functions with far fewer units than shallow ones, and the theorem's guarantee for one layer turned out to be the wrong comfort. But the result is the reason nobody since 1989 has seriously argued that neural networks are limited in what they can express.