ADALINE and the least-mean-squares rule

Two years after the perceptron, Bernard Widrow and his student Ted Hoff, who would later co-invent the microprocessor at Intel, built a different learning neuron at Stanford. ADALINE, the adaptive linear neuron, did not learn from whether its thresholded output was right or wrong. It learned from the size of the error before the threshold, and it moved each weight a small step in the direction that reduced the squared error. That is gradient descent, applied to a single neuron, and they called it the least-mean-squares rule.

The difference from Rosenblatt's rule looks small and is not. Because the LMS update is proportional to a smooth error, it can be analysed, it converges gracefully, and it extends to any model whose output is a differentiable function of its weights. The delta rule in the 1986 backpropagation paper is Widrow–Hoff with the chain rule attached.

ADALINE also worked. Built from memistors, electrochemical resistors whose value changed with use, it was among the first learning hardware to leave the laboratory: adaptive filters based on the LMS algorithm went into modems, echo cancellers and later every mobile phone. It is the most deployed learning algorithm of the twentieth century, and almost nobody outside signal processing knows its name.