SK hynix Details Memory-Tiering Technologies for AI Inference

SK hynix is developing two technologies to distribute AI inference KV cache across high-bandwidth memory (HBM), dynamic random-access memory (DRAM) and solid-state drives (SSDs), addressing capacity and cost limits as agentic AI and longer-context workloads expand. The technology update follows SK hynix’s record third-quarter operating profit.

KV cache is data AI systems retain and reuse during inference. SK hynix said the amount is growing as agentic AI and long-context applications spread. Keeping the entire cache in HBM or DRAM faces capacity and cost constraints. The company’s tiered approach allocates data among all three memory types based on access patterns.

One technology, SALT-KV (Semantic-Aware Lifecycle Tiering for KV), evaluates the reuse value of each cache item and its storage cost. It uses those factors to distribute data among HBM, DRAM and SSD, with the stated goal of controlling expense while maintaining inference efficiency. The design makes cache reuse value part of the placement decision alongside storage cost.

CMM-Hybrid uses a CXL-based design with a different storage arrangement. It keeps the full KV cache on SSD, predicts which data will be needed and moves it into DRAM before access. That preloading is intended to reduce subsequent access latency. The system retains the full cache on SSD and stages predicted data in DRAM.

The company also sees SSDs potentially serving as a context-memory tier, where multiple servers and GPUs could share large context caches. That would expand their role from KV cache offloading to holding context for reuse across systems. The approach reflects SK hynix’s effort to make NAND storage part of AI inference infrastructure as well as conventional data storage.

SK hynix is demonstrating the technologies with actual AI workloads to evaluate whether enterprise SSDs meet inference performance needs. The demonstrations are preparation for future adoption in customer and platform-partner system architectures. Together, the designs test two ways of using SSDs for inference cache: allocating data based on reuse value and cost, or retaining the full cache on SSD while bringing predicted data into DRAM.

For a longer-term tier, SK hynix plans to develop High Bandwidth Flash (HBF) between HBM and SSDs. The concept would combine high-bandwidth access with NAND’s capacity and cost efficiency, adding another memory level to the company’s approach for AI inference.