Playing Atari with Deep Reinforcement Learning
Mnih et al., 2013 (the "DQN" paper) — combined Q-learning with a neural network to learn to play Atari games directly from raw pixels, no hand-engineered features, from a single reward signal.
"Playing Atari with Deep Reinforcement Learning" (Volodymyr Mnih and colleagues at DeepMind, 2013) introduced the Deep Q-Network (DQN) — the paper that showed reinforcement learning could work directly from raw sensory input, not hand-designed features, using nothing but the score as a training signal.
What problem it solved
Classic Q-learningQ-LearningQ-learning learns the expected future reward of taking a specific action in a specific state, updating its estimate incrementally from every action the agent actually takes. stores a value for every (state, action) pair in a table — workable for small, discrete state spaces, but hopeless for anything with a visually rich environment, like a video game's raw pixel screen. There are far too many possible pixel configurations to ever build or store such a table, and no way to generalize from one state to a visually similar one it hasn't seen before. Prior attempts to combine neural networks with Q-learning existed but were notoriously unstable — the combination tended to diverge rather than converge to a useful policy.
The key idea
Replace the Q-value lookup table with a neural network that takes raw pixels as input and outputs an estimated value for each possible action — the same "swap a hand-designed representation for a learned one" pattern that runs throughout this course. Getting this to actually train stably leaned on one key technique:
- Experience replay: instead of training on each new experience once and discarding it, store experiences in a buffer and train on randomly sampled batches from it. This breaks the strong correlation between consecutive frames of gameplay, which otherwise destabilizes training the same way training on non-shuffled, ordered data would.
A second stabilizer often bundled into the name "DQN", a separate, slowly-updated target network used to compute training targets so they don't shift with every update, came in DeepMind's 2015 Nature follow-up, "Human-level control through deep reinforcement learning," which is also where the DQN name comes from.
Why it mattered
The network learned to play seven Atari 2600 games from nothing but raw pixel input and the game's own score, outperforming all earlier methods on six of them and a human expert on three, with no game-specific engineering: the same architecture, algorithm, and hyperparameters were used for every game. That was the first clear, widely-recognized demonstration that deep learning and reinforcement learning could be combined into something that scaled and actually worked, rather than being a theoretical curiosity — directly setting up the case, covered in Reinforcement Learning, for RL as a serious third paradigm alongside supervised and unsupervised learning, and for DeepMind's later, much larger results like AlphaGo.
Authors: Volodymyr Mnih, Koray Kavukcuoglu, David Silver, and colleagues (DeepMind)
Read the paper — arXiv:1312.5602Learn more: Q-LearningQ-LearningQ-learning learns the expected future reward of taking a specific action in a specific state, updating its estimate incrementally from every action the agent actually takes. · Reinforcement Learning
Denoising Diffusion Probabilistic Models
Ho, Jain & Abbeel, 2020 — showed that training a network to reverse a fixed noise-adding process, one small step at a time, could generate images matching the best GANs of the time.
Proximal Policy Optimization Algorithms
Schulman et al., 2017 — a policy-gradient method that keeps each update small by clipping how far the new policy may move, simple enough to become the default and the RL step inside RLHF.