MACaQ Quantizes a 122-Billion-Parameter Model in 64 Minutes

The MACaQ post-training method quantized a model with 122 billion parameters in 64 minutes, using 32.5 gigabytes of GPU memory. The approach targets lower memory and compute costs during neural-network inference.

MACaQ uses MXFP4 W4A4, a format that represents weights and activations at four-bit precision. Because quantization happens after training, it changes how a model’s values are represented for inference. Inference is the stage when a trained model processes inputs and produces outputs; lowering precision is intended to reduce the memory and computation needed at that stage.

The method handles each linear layer separately. It uses per-channel scaling and block-Hadamard transforms to process activations online, then incorporates inverse transforms into the weights. Its streaming design can process layers sequentially while keeping one layer in memory at a time during quantization. That one-layer-at-a-time figure applies to the quantization process.

MACaQ also uses Flip-Refinement, a Hessian-aware coordinate-descent procedure that adjusts FP4 codes to neighboring representable values. The refinement is part of the method’s effort to preserve model quality when weights and activations use four-bit values. The method is designed for different architectures and modalities, including text, image, video and audio models.

In benchmark comparisons, MACaQ recorded lower perplexity than DuQuant++ and MR-GPTQ. Perplexity measures how well a language model predicts text, with lower results indicating better performance on that measure. On Llama-3-8B, MACaQ matched the best perplexity result from the two comparison methods while using one-eleventh as much quantization time. The result applies to that model and benchmark.

In vLLM tests with Qwen3-32B, MACaQ added 2.3% end-to-end overhead relative to pure MXFP4. The implementation supports llama.cpp, vLLM, Diffusers and PyTorch. A companion version, MACaQ-R, refines existing GGUF models for specific workloads and remains compatible with standard llama.cpp.

The method adds to work on lower-precision model inference, including a separate GLM tuning effort using NVFP4 quantization.