Google Releases XLA Serving Engine to Deploy Models Without Python

Google released the XLA Serving Engine (XSE), a C++ runtime for deploying compiled machine-learning models without Python in the serving environment. The open-source release, announced Nov. 5 under the Apache 2.0 license, adds a model-serving option to Google’s OpenXLA ecosystem.

The workflow separates model development from deployment. Developers can train with JAX or PyTorch and export a standalone artifact. The runtime handles serving after compilation, so the deployed model does not need to carry the Python interpreter with it.

A single exported artifact can run on CPUs, NVIDIA GPUs and Google Cloud TPUs. Arm’s Neoverse work also extends XSE to Arm-based server chips. The hardware options give developers a way to serve compiled models across several processor types while using the same runtime.

XSE targets recommendation systems and vision applications.

Google uses XSE for workloads that include YouTube Shorts and Google Ads. The runtime fits into the serving stage of a larger model workflow: OpenXLA tools, including the XLA compiler and StableHLO, support compilation, while XSE runs the resulting model.

The broader OpenXLA effort dates to 2023 and provides a shared compilation layer for machine-learning frameworks and accelerators. XSE adds the deployment step after that compilation work, extending the ecosystem from preparing models for different hardware to serving them in production.

The release adds to Google Cloud’s earlier PyTorch-on-TPU serving work, which used Ray to coordinate training and model serving on Google TPUs. XSE is a separate C++ serving runtime within OpenXLA.