Deep Q-networks play Atari

Watkins's Q-learning stored a value for every state and action in a table, which was why it had never scaled. DeepMind's paper, posted on 19 December 2013 and published in Nature in February 2015, replaced the table with a convolutional network that took the last four frames of an Atari 2600 screen as input and produced a value for each joystick action. The network was trained on Watkins's update rule, with two additions that made it stable: a replay memory that shuffled past experiences so consecutive updates were not correlated, and a slowly updated copy of the network to provide the targets.

The same program, with the same settings, learned 49 games from nothing but the pixels and the score. On 29 of them it matched or beat a professional human tester. It discovered, in Breakout, the trick of tunnelling through the wall to bounce the ball behind it, which nobody had told it about.

The paper is the origin of deep reinforcement learning as a field and the reason Google bought DeepMind a month after the preprint appeared. Its recipe, a deep network trained by temporal-difference error from experience, scaled to AlphaGo two years later and to the agents of the 2020s that learn to use computers.