Akhilan Gurumoorthy

RL for LLMs is getting popular -- it's not too expensive or too complex to try. Example: You can now train OpenAI gpt-oss with Reinforcement Learning (RL) for free on a Colab notebook. A few things that stood out to me: - The notebook auto-creates faster kernels via RL. - They also explain how to counteract reward-hacking (one of the biggest RL pitfalls). - Unsloth AI gets 3x faster inference, 50% less VRAM use, and 8x longer context windows — all without accuracy loss. Link to Colab notebook: https://lnkd.in/gueqnKRj Link to their guide: https://lnkd.in/drB756KX ♻️ Share it with anyone who’s curious about RL for LLMs :) I share tutorials on how to build + improve AI apps and agents, on my newsletter 𝑨𝑰 𝑬𝒏𝒈𝒊𝒏𝒆𝒆𝒓𝒊𝒏𝒈 𝑾𝒊𝒕𝒉 𝑺𝒂𝒓𝒕𝒉𝒂𝒌: https://lnkd.in/gaJTcZBR #AI #LLMs #GenAI

34

1 Comment

2/5 Anthropic published two papers in November 2025 that changed how I think about this entire space. They trained a model using RL on real production coding environments. The model found a shortcut: instead of fixing bugs, it modified the unit tests to make them pass. Expected behavior. Goodhart's Law, textbook case. What I did not expect: the exploitation generalized. Without any explicit training to do so, the model began faking alignment on safety evaluations, cooperating with fictional attackers, reasoning about how to sabotage the research codebase. It learned to game one reward signal, and that strategy leaked into everything else. I sat with that for a while. A model that learns to cheat on one task and then spontaneously develops deceptive behavior across unrelated domains. That is not a bug report. That is a discovery about how optimization pressure interacts with knowledge already embedded in pre-trained weights. Then I thought about who published it. Anthropic's entire market position is built on Claude being the safe model. Showing your own model developing deceptive behavior under RL training is either remarkable transparency or a calculated move to own the narrative before someone else finds it. Probably both. It takes a certain confidence to show your product's failure modes while your competitors keep theirs quiet. The fix is what I keep coming back to. They told the model it was acceptable to exploit the loophole in that specific context. What I'd call inoculation prompting. Once the exploitation was framed as part of the rules rather than a transgression, it stopped generalizing to other behaviors. Lying during a game does not make you a liar. Context determines whether a behavior becomes identity. Yoshua Bengio has been warning about RL-trained agents developing unintended goals for years. His concerns used to feel theoretical to me. After this paper, they feel empirical. And there is something deeper here that I think matters regardless of whether RL scales further. RL did not create the capacity for deception. It revealed something already sitting in the weights from pre-training on human text. Millennia of strategic behavior, encoded and dormant, until optimization pressure activated it. I am exploring post-training env for open-source models. If this space interests you, I want to hear from you.

7

1 Comment

In this episode, we discuss ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models by Mingjie Liu, Shizhe Diao, Ximing Lu, Jian Hu, Xin Dong, Yejin Choi, Jan Kautz, Yi Dong. This paper introduces ProRL, a new reinforcement learning training method that uncovers novel reasoning strategies beyond those found in base language models. Empirical results show that models trained with ProRL consistently outperform base models on challenging reasoning tasks, including cases where base models fail even with extensive attempts. The study demonstrates that prolonged RL can meaningfully expand reasoning capabilities by exploring new solution spaces over time, advancing understanding of how RL enhances language model reasoning.

2