# Lila Sciences and Anyscale Train GLM-5.3-Flash on 1 Million Tokens

By Simon Yoon

Canonical URL: https://www.tokenpost.com/news/technology/50644
Published: 2026-11-07T06:51:21.000Z
Updated: 2026-11-07T06:51:21.000Z
Section: Technology

> A separate human-genome test at 512,000 tokens per sequence recorded a 4.8% drop in held-out loss over 80 steps.

Lila Sciences and Anyscale trained GLM-5.3-Flash on sequences of up to 1 million tokens using 64 NVIDIA B200 GPUs, extending its reach for scientific datasets such as human genomes.

The run used SkyRL, an open-source training stack, and marked a 64-fold increase over the initial training capacity. At 1 million tokens, throughput was 13,700 tokens per second and peak memory use reached 144.5 GiB per GPU. Throughput peaked at 24,900 tokens per second at 256,000 tokens.

A separate fine-tuning test used human-genome sequences of 512,000 tokens. Chromosomes 1 and 2 were used for training, while chromosome 22 was reserved for evaluation. Held-out loss fell 4.8% over 80 steps, with the largest gains on tokens that had the most preceding context.

GLM-5.3-Flash combines 34 layers of Kimi Delta Attention with 11 layers of DeepSeek Sparse Attention and includes mixture-of-experts components. Changes to tensor dimensions enabled sparse-attention kernels for the DeepSeek layers and cut memory use for that component by half. The training also used activation recomputation, CPU offloading and context parallelism to manage memory and processing demands.

At 256,000 tokens, offloading lowered peak memory use from 142 GiB to 98 GiB, while increasing step time by 21%. Routing real data through the mixture-of-experts layers produced an uneven load: the busiest GPU handled up to four times the average number of tokens. Chunking those layers addressed memory bottlenecks at 512,000 tokens.

At 1 million tokens, offloaded layer inputs needed about 180 GB of CPU memory per GPU. The setup used pinned memory to speed transfers and released memory after each training step. Validation also included reinforcement-learning runs and comparisons with vLLM, with a log-probability gap of 0.04 to 0.07 across context lengths from 128,000 to 1 million tokens.

All project changes are open source and integrated into SkyRL. Several upstream patches were submitted to Megatron-LM.