Rectified linear units

For two decades the standard artificial neuron squashed its input through a sigmoid, a smooth S-curve that saturates at both ends. The saturation is where the gradient dies: a unit that is strongly on or strongly off passes almost no error signal back. Vinod Nair and Geoffrey Hinton's ICML paper in June 2010 tried the simplest possible alternative, output the input if it is positive and zero otherwise, and found that it worked better in restricted Boltzmann machines. Xavier Glorot, Antoine Bordes and Yoshua Bengio showed the next year that it worked better in deep supervised networks too, with no pretraining needed.

The rectifier's derivative is one for any active unit, so the gradient passes through undiminished however deep the network, and half the units are exactly zero at any time, which is cheap and sparse. It is a McCulloch–Pitts threshold with a linear ramp above it, and the field had walked past it for sixty years because it was not differentiable at zero and did not look like a neuron.

AlexNet used it in 2012 and reported training six times faster than with sigmoids. Almost every network since has used it or a close relative. Along with GPUs, data and dropout, it is one of the four ingredients that made 2012 possible.