A Photonic-CXL Memory Appliance for Scalable KV Cache Management in LLM Inference

2026/07/29 by Jing Ding, Yash Nishant, Chandrish Ambati +2
Computer Science · #cs.PF

paper · pdf

arxiv created 2026/07/29 · arxiv updated 2026/07/30

Abstract

LLM inference at scale faces a memory wall. The KV cache demands tens of terabytes at hundreds of gigabytes per second, yet no current memory tier delivers both at once. Characterization across multi-generation GPU systems with various LLaMA models shows host memory retrieval achieves up to 100x speedup over re-computation but supports only tens of concurrent long-context users. Electrical CXL pooling theoretically bridges this gap, but switch latency, cable reach limits, and power-scaling issues prevent practical TB-scale deployments. We present the Marvell Photonic Fabric Memory Appliance, a photonic-CXL hybrid architecture replacing electrical switches with a passive fiber shuffle to deliver 32 TB shared memory across 16 hosts via a switch-free full- crossbar topology. Emulation results demonstrate over 50 percent latency reduction versus electrical CXL pools. Simulation results show that the PF Memory Appliance eliminates cache eviction cliffs by improving time-to-first-token by 6.6x for multi-turn conversations workloads.

Related

• Beluga: A CXL-Based Memory Architecture for Scalable and Efficient LLM KVCache Management
• Scalable Processing-Near-Memory for 1M-Token LLM Inference: CXL-Enabled KV-Cache Management Beyond GPU Limits
• HyMCache: A KV Cache Framework for Multi-Turn LLM Serving with CXL-Hybrid Memory
• LiteCache: A Query Similarity-Driven, GPU-Centric KVCache Subsystem for Efficient LLM Inference
• Photonic Fabric Platform for AI Accelerators