IBM-Backed llm-d 1.0 Adds Controls for AI Inference Workloads

The llm-d project released version 1.0 on Oct. 20, adding workload controls and routing features for organizations running large AI models on Kubernetes infrastructure they manage.

IBM, Red Hat and Google are lead maintainers of llm-d, an open-source framework that distributes inference requests among model servers such as vLLM and SGLang. The project is hosted by the Cloud Native Computing Foundation as a sandbox project.

The new controls let operators prioritize requests, share capacity among users and workloads, and shed traffic when demand exceeds capacity. Existing deployments keep their current scheduling behavior unless operators enable the new policies. Version 1.0 also adds a configuration API.

The release includes a standby router. Active-active routing and failover for individual inference requests remain planned features. llm-d also supports coordinated upgrades where separate machine groups handle prompt processing and response generation.

Session-aware routing can send a later request from an AI agent to a server that holds context from an earlier request. Reusing that context reduces repeated processing. The feature is designed for agents that make multiple calls during a task.

A benchmark published Sept. 8, before version 1.0, ran 3,000 concurrent coding agents on 544 NVIDIA H100 GPUs. The deployment generated 6.60 million output tokens per minute at that load, near its serving limit.

Carlos Costa, an IBM Research distinguished engineer and llm-d co-founder, described the project’s goals as “a stable API, multiple tenants with competing priorities sharing the same infrastructure, and upgrades that don't interrupt operations.”