Chunk-Aware KV Cache Management Bridges LLM Serving and CXL-SSDs

This episode examines why a stock CXL-attached SSD, meaning flash capacity exposed as load/store memory behind a small device-side DRAM cache, serves reused prompt prefixes about three times slower than local DRAM. It explains how prefix caching cuts time-to-first-token by skipping prefill, and how LMCache stores KV in fixed-size token chunks across GPU, host DRAM, and storage tiers. A ramdisk experiment on Qwen3-4B with a 34k-token prefix shows the block I/O interface alone costs 1.8x in TTFT, through staging copies and last-level cache contention, even with no flash involved. A generic next-n prefetcher in the device also fails, because the device sees only page addresses while the serving engine knows which chunks a request needs and in what order. That semantic gap motivates the paper's redesign around the chunk as the shared unit between engine and device, and the episode compares it briefly with the concurrent HyMCache and ITME work.

Sources:

1. Bridging LLM Serving and CXL-SSDs with Chunk-Aware KV Cache Management — Hyunsun Chung, Taewan Noh, Minji Kim, Joo-Young Hwang, Hong-Yeon Kim, Youngjae Kim, 2026

http://arxiv.org/abs/2609.26828v1

2. Pond: CXL-Based Memory Pooling Systems for Cloud Platforms — Huaicheng Li, Daniel S. Berger, Lisa Hsu, Daniel Ernst, Pantea Zardoshti, Stanko Novakovic, Monish Shah, Samir Rajadnya, Scott Lee, Ishwar Agarwal, Mark D. Hill, Marcus Fontoura, Ricardo Bianchini, 2023

https://scholar.google.com/scholar?q=Pond%3A+CXL-Based+Memory+Pooling+Systems+for+Cloud+Platforms

3. TPP: Transparent Page Placement for CXL-Enabled Tiered-Memory — Hasan Al Maruf, Hao Wang, Abhishek Dhanotia, Johannes Weiner, Niket Agarwal, Pallab Bhattacharya, Chris Petersen, Mosharaf Chowdhury, Shobhit Kanaujia, Prakash Chauhan, 2023

https://scholar.google.com/scholar?q=TPP%3A+Transparent+Page+Placement+for+CXL-Enabled+Tiered-Memory

4. Efficient Memory Management for Large Language Model Serving with PagedAttention — Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, Ion Stoica, 2023

https://scholar.google.com/scholar?q=Efficient+Memory+Management+for+Large+Language+Model+Serving+with+PagedAttention

5. CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion, and the LMCache system line (LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference) — Jiayi Yao, Hanchen Li, Yuhan Liu, Siddhant Ray, Yihua Cheng, Qizheng Zhang, Kuntai Du, Shan Lu, Junch