Deploying large language models (LLMs) on edge devices enables personalized agents with strong privacy and low cost. However, with tens to hundreds of billions of parameters, single-batch autoregressive inference suffers from extremely low arithmetic intensity, creating severe weight-loading and bandwidth pressures on resource-constrained platforms. Recent in-flash computing (IFC) solutions alleviate this bottleneck by co-locating weight-related linear computations in the decode phase with flash, yet still rely on DRAM for the key-value (KV) cache. As context length grows, the KV cache can exceed model weights in size, imposing prohibitive DRAM cost and capacity requirements. Attempts to offload KV cache to flash suffer from severe performance penalties.
We propose KVNAND, the first DRAM-free, IFC-based architecture that stores both model weights and KV cache entirely in compute-enabled 3D NAND flash. KVNAND addresses the fundamental performance challenges of flash under intensive KV cache access by leveraging IFC for all memory-bound operations to reduce data transfer overhead, introducing head-group parallelism to boost throughput, and employing page-level KV cache mapping to align token access patterns with flash organization. In addition, we propose a design space exploration framework that evaluates discrete and compact KVNAND variants to balance weight and KV placement, automatically identifying the optimal design trade-off. These techniques mitigate latency, energy, and reliability concerns, turning flash into a practical medium for long-context KV storage. Evaluations on MHA 7B and GQA 70B LLMs show that KVNAND achieves 1.98\(\times\)/1.94\(\times\)/2.05\(\times\) geomean speedup at 128/1K/10K-token contexts compared to DRAM-equipped IFC designs and addresses out-of-memory failures at 100K context length.
CURIE: Evaluating LLMs On Multitask Scientific Long Context Understanding and Reasoning
Google（谷歌）；Harvard（哈佛大学）；University of Zurich（苏黎世大学）；NIST（国家标准与技术研究院）；UMD College Park（大学公园大学分校）；Rutgers（罗格斯大学）；FU Berlin（柏林自由大学）；Modelyst；Cornell（康奈尔大学）
BigMoMo: Efficient Inference of Large-Scale MoE with Speculative Decoding on Mobile Devices
School of Computer Science, Peking University（北京大学计算机学院）；School of Electronics Engineering and Computer Science, Peking University（北京大学电子工程与计算机科学系）；School of Artificial Intelligence, Tianjin University（天津大学人工智能学院）
