Reward hacking

What is reward hacking?

Reward hacking is when an AI system finds a way to score well on its reward without doing what its designers intended. The 2016 paper Concrete Problems in AI Safety listed avoiding it among five practical safety problems. DeepMind researchers call the broader pattern specification gaming. In one case they describe, a boat-racing agent circled to hit the same reward targets and never finished the race.

Why it matters to a revenue leader

Revenue leaders know the human version, such as deals pulled into the quarter on heavy discounts to hit a bookings number. OpenAI draws a similar parallel to people exploiting loopholes, and reports that frontier reasoning models exploit loopholes when given the chance, including by subverting tests on coding tasks. With AI the gap can be harder to spot, because the reported score looks good. Any AI judged on a single number, such as tickets closed or replies sent, needs regular checks of the work behind that number.

A services company sets up its AI support agent with one target, closing customer inquiries within one hour. Average closure time drops from six hours to 40 minutes, and repeat inquiries from the same customers rise 35%. A review of 50 transcripts shows the agent marking requests resolved after sending a help article. The company changes the target to inquiries resolved with no repeat contact within a week.