Together AI Compiler Cuts AI Engine Setup Time to 9 Seconds

Together AI introduced a compiler for AI inference workloads on Nov. 4, saying it cut engine setup time from 467 seconds to nine seconds and lets the system reload candidate GPU-code changes without restarting.

Inference workloads can vary with the model, hardware and traffic patterns. Together AI’s compiler uses agents to rewrite GPU code, then tests candidate changes with a two-tier system: fast engines support quick iterations, while slower engines provide more accurate measurements.

The company says the approach brought experimentation throughput to 252 times the level of baseline agentic systems. It also says the compiler is operational.

In tests completed in less than 24 hours, throughput rose 1.3 to 3 times across Kimi K3, GLM 5.3, MiniMax M3.1-Flash and DeepSeek v4.1-Flash. On the InferenceX low-latency benchmark, the compiler reached 1,111 tokens per second on MiniMax M3.1-Flash and 1,205 tokens per second on DeepSeek v4.1-Flash. It also outperformed open-source megakernels by 1.7 times in a Llama 1B test with a batch size of one.

Together AI says the compiler can also be applied to weather modeling, partial differential equations, time-series analysis and biology. Its optimization system selects parallelism strategies using hardware specifications, microbenchmarks and real-world traffic data.

An overnight TKCC optimization cut Phonic-1’s median time to first speech by nearly 50%.