Researchers Propose Two-Reward Method for More Stable AI Training

Shourov Joarder and three co-authors propose a way to train language models with two forms of self-generated feedback, aiming to make reinforcement learning more stable without external supervision.

The approach uses reinforcement learning from internal feedback (RLIF), in which a model generates its own training signals. By comparison, reinforcement learning with verifiable rewards (RLVR) often uses human annotations or gold-standard solutions. The authors say RLIF methods that rely on a single reward can be vulnerable to reward hacking, entropy collapse and weakened reasoning structure.

The proposed method pairs an answer-level signal based on cluster voting with a completion-level signal based on token-by-token self-certainty. GDPO-based normalization balances the scales of the two rewards. A separate technique, KL-Cov regularization, targets low-entropy token distributions. The authors say it is intended to preserve exploration and prevent collapse late in training.

On mathematical reasoning and code-generation benchmarks, the authors report greater stability and robustness than earlier unsupervised reinforcement-learning approaches. They say the results approach those of supervised RLVR. The authors plan to release the code.