TokenPost topic
Inference
6 TokenPost stories are tagged Inference. The newest appear first.
Latest Inference stories
- Super Micro Computer, IBM and Nvidia Introduce KV Cache Solution
The solution showed significant performance advantages during inference. A KV cache holds intermediate calculations a model uses while processing input.
· Technology
- NVIDIA Details TensorRT Tuner’s 1.14x to 1.63x Performance Across 10 Workloads
The Global Performance Tuner option was added to `trtexec` in TensorRT 11.1, released June 24, 2026.
· Technology
- Cerebras Executive Says Inference Speed Matters for AI Agents
Andy Hock argued that speed becomes more consequential when one request sets multiple agents to work. Cerebras’ up-to-15-times claim applies to selected open models, not full workflows.
· Technology
- Anthropic API Gross Margin Estimated Above 80% as Inference Margin Reaches Mid-60s
Subscriptions account for about 10% of revenue but use more than 40% of inference computing capacity, lowering blended revenue per megawatt by approximately $36 million.
· Business
- Microsoft AI CEO Says Frontier Model Training Could Cost $100 Billion
Mustafa Suleyman said five or six labs, including Microsoft AI, can assemble gigawatts of computing power. He also said inference costs fell 300-fold over two years.
· Technology
- Crusoe’s Inference ARR Tops $100 Million as Firmus Plans $5 Billion IPO
A $65 million annual agreement accounts for about 60% of Crusoe’s inference contract ARR, while Firmus expects a $77 million fiscal first-half loss.
· Business