Unlocking Agentic RL Training for GPT-OSS: A Practical Retrospective

Agentic reinforcement learning (RL) optimizes a model's entire decision-making process through environmental interaction, unlike traditional single-turn training. This retrospective from LinkedIn details the practical challenges of applying agentic RL to open-source GPT models (GPT-OSS). The team encountered issues like exploding KL divergence and stagnant rewards, which they traced to problems in the training framework and model architecture. Key solutions involved restoring on-policy integrity for PPO, correcting training-inference mismatches, and implementing memory-efficient techniques like attention sink support in FlashAttentionV3.