VIMPO beats GRPO on hard math benchmarks without training a critic

Reinforcement learning with verifiable rewards (RLVR) is how teams squeeze more reasoning out of math-tuned LLMs. The usual fork:

• Group-relative methods (GRPO) — no critic network, but trajectory-level advantages hit every token the same way
• Actor-critic (PPO-style) — finer credit assignment, plus the pain of training and stabilizing a value head

A new method called VIMPO (Value-Implicit Policy Optimization) claims a third path: critic-free like GRPO, but with denser learning signals derived from the policy itself.

On competition-style math evals, VIMPO beats GRPO with especially large gains on harder sets like AIME 2025. Under noisy rewards it keeps a consistent edge. If you fine-tune reasoning models, this is worth a read before your next GRPO run.

The GRPO credit assignment problem

VIMPO paper frames the trade-off clearly. GRPO avoids learning a separate value network. That saves complexity. The cost is coarse advantages: one scalar reward shapes every token in the trajectory equally.

For short answers that might be fine. For long chain-of-thought reasoning, early tokens that set up a wrong path get the same gradient nudge as tokens that salvage the proof. Actor-critic methods fix that with per-step values, but value networks bring their own instability and hyperparameter surface.

VIMPO asks: can we get implicit values from the policy and reference model without training a critic?

How VIMPO builds implicit values

The derivation starts from KL-regularized RL optimality conditions. For autoregressive generation, the value recurrence can be written using policy-reference log-ratios, anchored by a terminal condition: no future reward remains at sequence end.

That yields:

1. A value loss that incorporates outcome-level verifiable rewards without a learned critic
2. A critic-free actor advantage for a PPO-style policy update

Same practical simplicity as GRPO. Finer credit assignment than uniform trajectory advantages.

Code: github.com/backprop07/VIMPO

Benchmark results (reported)

Paper tables compare VIMPO against GRPO variants on mathematical RLVR benchmarks. Headline numbers from the abstract and results discussion:

Exact configs depend on base model and training budget. Treat these as directional unless you reproduce on your stack.

The authors highlight noisy reward settings: VIMPO retains advantage over GRPO when verifiers are imperfect. That matters for real pipelines where unit tests pass but proofs are shaky.

Related work: single-rollout critic-free RL

The same week, Rethinking Groups in Critic-Free RLVR (arXiv:2606.17250) attacks a different GRPO pain point: group size. GRPO often needs many rollouts per prompt (G=16 in their tables) for stable relative advantages.

They introduce Negative Token Filtering (NTF) so C-RF with G=1 trains stably. On Qwen2.5-Math-1.5B:

Single-rollout training approaching multi-rollout GRPO is a compute win. Different mechanism than VIMPO, same theme: critic-free RLVR is still under-optimized.

When I would experiment with VIMPO

Reproduction checklist:

1. Match base model and training tokens to your GRPO baseline
2. Hold verifier constant (same unit tests / sympy checks)
3. Evaluate on held-out competition sets, not just training-adjacent MATH subsets
4. Track wall-clock and rollout cost, not just accuracy

Practical takeaway for applied teams

Most consulting clients are not training Qwen-Math from scratch. But the pattern propagates:

Verifier-driven RL is the default for reasoning fine-tunes. Advantage estimation is still the lever.

When a vendor says "we use GRPO," ask how advantages attach to tokens and how many rollouts per prompt you pay for. VIMPO and NTF-style papers are evidence that default GRPO is not the ceiling.

If you are building custom model eval or fine-tune pipelines and want help choosing RLVR setup for your domain (code, ops workflows, structured extraction), book a call.

Sources

• VIMPO paper
• VIMPO code
• Rethinking Groups in Critic-Free RLVR (NTF)