0

To Critic or Not to Critic: GRPO vs PPO

https://labs.scale.com/blog/to-critic-or-not-to-critic-grpo-vs-ppo(labs.scale.com)
An extensive empirical study compares two reinforcement learning algorithms: Proximal Policy Optimization (PPO), which uses a critic, and Group Relative Policy Optimization (GRPO), which does not. The research investigates when and under what conditions a learned critic improves training efficiency across various tasks, including control, math reasoning, and long-horizon agents. Results indicate that PPO generally performs better at small batch sizes, while GRPO's performance improves and often surpasses PPO as the batch size increases. This crossover point is pushed to larger batch sizes for more complex tasks. In terms of compute efficiency, PPO is more effective on small budgets, whereas GRPO can match or exceed PPO's performance with less compute on larger budgets.
0 points•by hdt•18 hours ago

Comments (0)

No comments yet. Be the first to comment!

Have an account? Log in to join the discussion.