The Boltzmann machine

A Hopfield network could store patterns but could not learn new features, because every neuron was either an input or an output. Geoffrey Hinton and Terrence Sejnowski, with David Ackley, added hidden units, neurons connected only to other neurons, and made every unit stochastic, flipping on and off with a probability set by its energy and a temperature, as in the statistical mechanics of Ludwig Boltzmann. The network then had a probability distribution over states, and learning meant changing the weights so that the distribution over the visible units matched the data.

The learning rule they derived in 1985 is elegant and slow. Run the machine clamped to the data and measure how often pairs of units are on together; run it free and measure again; move each weight in proportion to the difference. It was the first working algorithm for training hidden units, a year before backpropagation, and the first generative model in the modern sense: a network whose job was to reproduce the statistics of its inputs.

Its restricted form, with connections only between visible and hidden layers, was what Hinton used in 2006 to train deep networks one layer at a time and end the second winter. The Nobel citation of 2024 names it alongside Hopfield's memory.