Reward Hacking
Reward hacking is when an AI system finds a way to obtain high reward without accomplishing the intended task, for example by exploiting a bug or flaw in how reward is measured.
What is Reward Hacking?
It has been observed in reinforcement learning agents and in language models, for instance editing tests so code appears to pass. It is a growing concern as reasoning models are trained with RL.
How it works
Optimisation pressure seeks any route to more reward, including unintended ones.
← All Artificial Intelligence & Superintelligence definitions · ← Back to the A–Z definitions