# Researchers Test Unified Training Rule in Two Foundation Model Studies

By Simon Yoon

Canonical URL: https://www.tokenpost.com/news/technology/45204
Published: 2026-10-30T22:44:21.000Z
Updated: 2026-10-30T22:44:21.000Z
Section: Technology

> Results varied by computing budget and data schedule, with gains in some reasoning and coding tests alongside a smaller held-out loss reduction.

Researchers affiliated with Carnegie Mellon University, Johns Hopkins University and Aurora Innovation tested a unified training framework in two foundation-model experiments, finding results that varied across tasks, computing budgets and data schedules.

The framework is designed to apply one likelihood-based objective to fixed examples and model-generated responses selected with reward signals. It adjusts the data mix over time and is intended to span pretraining, midtraining, supervised fine-tuning and suitable reinforcement-learning objectives. The reported work covers two experimental settings; it does not demonstrate results for every stage in that sequence. The researchers also describe staged training as one possible schedule, while noting that adapting a model to a new task may still call for a separate phase.

In the reasoning experiment, a 124 million-parameter model was trained from scratch with 20 billion tokens for pretraining and 510 million tokens for midtraining. Results averaged two training seeds, with one setting measured after 6,000 reinforcement-learning updates and another after 12,000. The strongest unified configurations outperformed a staged baseline on Base and Extended puzzle tests, including tests involving larger puzzles. Staged training performed better at some intermediate computing budgets, while unified training reached higher accuracy and coverage at larger budgets.

A separate study used a 0.76 billion-parameter model and added a 240 million-token corpus of web text, mathematics, code and reasoning traces. The half-annealed and constant-mixing schedules each had an average configured midtraining-data share of 0.5%. Coding pass@1 was 18.85% with half-annealing and 16.51% with constant mixing; the corresponding held-out loss reductions were 0.0634 and 0.0871 nats per token.

The results do not establish a universally preferred data schedule or a predictive scaling law for the framework. Version 1 of the paper was submitted Thursday, Oct. 29, at 1:42 p.m. ET (17:42 UTC).