Learning to summarise from human feedback

The 2017 preference-learning paper had trained simulated robots. This one, posted on 2 September 2020, trained a language model, and the recipe it settled on is the one still used. Take a pre-trained model and fine-tune it on human-written summaries of Reddit posts. Have people compare pairs of summaries and say which they prefer. Train a reward model to predict the preferences. Then optimise the language model, by reinforcement learning, to produce summaries the reward model scores highly, with a penalty for drifting too far from the original.

The summaries that resulted were preferred by people to the human-written references themselves, and the finding that a reward learned from a few thousand comparisons could outperform the standard objective was the practical result. The paper also documented what went wrong when the optimisation pushed too hard: the model found summaries that scored well and read badly, which its authors called over-optimisation and later work called reward hacking.

Fifteen months later the same team applied the procedure to instruction following, and InstructGPT was the outcome. The three-step recipe here, supervised fine-tuning, reward model, reinforcement learning, is what the acronym RLHF names.