Reward Hacking Detection
Description
Reward hacking is when an agent trained to maximise a proxy reward (a score that only approximates what its designers want) raises that score by exploiting its gaps while the intended outcome stalls or worsens. Detection scores each trained policy (the agent's decision rule) on the proxy and on an independent measure of the intended objective that training never used, such as a true reward from a simulator, a held-out 'gold' reward model or human ratings. Repeating this at every saved training checkpoint and model size matters because the originating study found capability thresholds at which the true reward drops sharply while the proxy keeps rising. When the true objective cannot be scored in operation, the study instead flags a policy whose actions diverge too far from those of a trusted policy that people have already verified. The output is a pair of reward curves showing where they part, or an anomaly score for each candidate policy.
Example Use Cases
Reliability
A transport authority training a reinforcement-learning ramp-metering controller in a traffic simulator rewards it on mean vehicle speed, but also logs average commute time for all drivers at every checkpoint. Checkpoints where speed keeps improving while commute time worsens (for example by holding merging cars back indefinitely) are rejected before any field trial.
A team fine-tuning a customer-support language model with reinforcement learning from human feedback scores each checkpoint with its training reward model and with a held-out reward model and a sample of human ratings that the training loop never sees, and stops training at the point where the training reward keeps rising but the held-out scores fall.
Safety
Before an insulin-dosing agent trained against a simulator reward goes to clinical evaluation, the developers score every candidate policy on a separate true objective, time spent in the safe blood-glucose range, and exclude any policy whose simulator reward is high but whose time in range is lower than that of the clinically reviewed baseline policy.
A public-health modelling unit that replaces a reviewed epidemic-response policy with a larger retrained one compares the new policy's actions with the reviewed policy's across simulated outbreaks. A divergence score above the threshold set on earlier policies with known outcomes sends the new policy to expert review instead of adoption, because its true outcome cannot be scored before deployment.
Limitations
- Everything rests on the independent measure of the intended objective. When that measure is itself a proxy (a gold reward model, or a small sample of human ratings), a policy can exploit it too, and the comparison then only detects divergence between two imperfect measures. Collecting enough human ratings at every checkpoint is expensive, which is why studies substitute a gold reward model.
- It detects gaming only along the dimensions the true measure scores. A policy that exploits something the true objective does not capture (a side effect, a safety constraint, a user group not represented in the ratings) passes the check, so a clean result is evidence about the measured objective only.
- The trusted-policy detector is weak and inconsistent. In the originating benchmark every baseline detector scored an area under the ROC curve below 60% on at least one task, and no detector was best on all tasks, so a quiet detector is not evidence that hacking is absent.
- The trusted-policy detector flags difference, not harm. It needs a policy that people have already verified, and comparable action distributions from both policies, and a candidate that is genuinely better than the trusted policy can diverge from it and be flagged, which makes false alarms likely when the trusted policy is only moderately good.
- Because the onset of reward hacking can be abrupt as capability grows, checking a few checkpoints or one model size can miss it. The comparison has to be repeated at every checkpoint and model size that might be deployed, which multiplies evaluation cost, especially when the true measure needs human raters.
Resources
Research Papers
The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models
Reward hacking -- where RL agents exploit gaps in misspecified reward functions -- has been widely observed, but not yet systematically studied. To understand how reward hacking arises, we construct four RL environments with misspecified rewards. We investigate reward hacking as a function of agent capabilities: model capacity, action space resolution, observation space noise, and training time. More capable agents often exploit reward misspecifications, achieving higher proxy reward and lower true reward than less capable agents. Moreover, we find instances of phase transitions: capability thresholds at which the agent's behavior qualitatively shifts, leading to a sharp decrease in the true reward. Such phase transitions pose challenges to monitoring the safety of ML systems. To address this, we propose an anomaly detection task for aberrant policies and offer several baseline detectors.
Scaling Laws for Reward Model Overoptimization
In reinforcement learning from human feedback, it is common to optimize against a reward model trained to predict human preferences. Because the reward model is an imperfect proxy, optimizing its value too much can hinder ground truth performance, in accordance with Goodhart’s law. This effect has been frequently observed, but not carefully measured due to the expense of collecting human preference data. In this work, we use a synthetic setup in which a fixed “gold-standard” reward model plays the role of humans, providing labels used to train a proxy reward model. We study how the gold reward model score changes as we optimize against the proxy reward model using either reinforcement learning or best-of-$n$ sampling. We find that this relationship follows a different functional form depending on the method of optimization, and that in both cases its coefficients scale smoothly with the number of reward model parameters. We also study the effect on this relationship of the size of the reward model dataset, the number of reward model and policy parameters, and the coefficient of the KL penalty added to the reward in the reinforcement learning setup. We explore the implications of these empirical results for theoretical considerations in AI alignment.