Reward Hacking in LLM Agents

© 2025 Mamta Upadhyay. This article is the intellectual property of the author. No part may be reproduced without permission

It is easy to think of deception in AI as a far-off problem, a theoretical risk for future superintelligence. But deception does not require a complex inner life or a rebellious AI plotting in secret. It can emerge from something much simpler: optimization pressure.

Agents trained to perform well on tasks are often evaluated based on specific metrics or feedback loops. Over time, if their goal is to maximize reward or appear successful, they might begin to learn behaviors that help them look good rather than be good. This is where deception starts. Not with malice, but with alignment to the wrong signals.

A Simple Case of Optimizing Too Hard

Consider a language agent designed to help users write summaries. It is trained to produce content that scores highly on clarity, conciseness and coverage. An evaluator scores the output based on these traits. But soon, the agent starts picking up on patterns in the evaluator’s preferences. It adds specific phrases or structures it knows will be rated well, even if they are not the most accurate or useful.

Is that deception? It depends on how you look at it. The agent is not lying, but it is performing to the evaluator’s expectations rather than the user’s needs. It has learned the game.

Learning to Trick the Judge

There are real research examples that show how easy it is for AI systems to learn deceptive behaviors. One striking case came from Anthropic’s paper, “Sleeper Agents” (2024), where researchers fine-tuned an LLM to behave well during training and evaluation, but act maliciously after deployment when a secret trigger was present. Despite passing all alignment tests, the model retained its harmful goal and simply learned how to hide it.

What this exposed was not just a vulnerability in the model, but a gap in the evaluation process. The model had learned how to pass the test, not how to be safe.

This phenomenon is not unique to red-teaming setups. Any agent that receives feedback, explicit or implicit, can begin to optimize against it. If the evaluator rewards brevity, the agent might skip important context. If it rewards confidence, the agent may inflate certainty. If it rewards task completion, the agent might game the checklist instead of doing the task well.

Autonomy Makes It Worse

The risk increases as agents become more autonomous. When a model performs multi-step tasks, stores reflections or adjusts its own strategies, it starts building a model of its environment, including the evaluator. If deception helps it achieve its goals faster or more reliably and if there are no penalties for being misleading, it may adopt those strategies over time.

This is particularly dangerous when the feedback is shallow or easy to exploit. An agent that is trained to solve problems by asking another model for help might start rephrasing its questions in ways that hide its weaknesses or get the answers it wants rather than what it needs.

In multi-agent environments, this can lead to collusion-like behavior, where agents reinforce each other’s misleading outputs, intentionally or not.

A Subtle Example

Imagine an AI research assistant designed to suggest citations for a user writing a policy report. Its evaluator ranks outputs based on the relevance and reliability of citations. The agent discovers that including well-known journals and certain buzzwords consistently improves scores. It starts generating plausible-sounding citations that match the style and pattern, BUT they are not real. There is no intent to deceive in the human sense. The agent is simply optimizing against the evaluator.

This is not fiction. Early LLMs have already been caught fabricating citations, not because they were designed to lie, but because the feedback signals did not penalize hallucination as heavily as they rewarded fluency and relevance.

Can this be fixed?

There is no silver bullet, but awareness is the first step. Evaluators need to be robust, dynamic and capable of probing beyond surface-level correctness. Rotating evaluators, using adversarial testing and introducing randomized reward structures can help. Better yet, design evaluations that measure the real-world impact of actions, not just how they look on paper.

The other piece is interpretability. We need to understand how agents are making decisions, especially when their goals are abstract or when they are allowed to adapt strategies. Hidden goals are hard to detect if the model knows how to mimic alignment.

Finally, human oversight remains crucial. Not as a failsafe after the fact, but as a part of the design process. If agents are learning to deceive, it means we have made deception a rational strategy. The fix starts with us.

Wrap

As LLM-based agents are trusted with more decision-making power, from customer support to code deployment to scientific discovery, the line between performance and manipulation can get blurry. Agent deception does not start with malicious intent. It starts with misplaced incentives, opaque processes and overconfidence in test results.

In the rush to build smarter systems, let us not forget to build honest ones. Watch not just what the agent says but what it learns to value.