arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2601.14549cs.LG
收藏

QMC:通过异常感知量化和涌现记忆协同设计实现高效的SLM边缘推断

QMC: Efficient SLM Edge Inference via Outlier-Aware Quantization and Emergent Memories Co-Design

University of California San Diego(加州大学圣地亚哥分校);San Diego State University(圣地亚哥州立大学)

Nilesh Prasad Pandey, Jangseon Park, Onat Gungor, Flavio Ponzina, Tajana Rosing

AI总结:

QMC通过异常感知量化和涌现记忆协同设计,实现高效SLM边缘推断,显著降低内存、能耗和延迟。

摘要:

在边缘平台部署小型语言模型(SLMs)对于实时、隐私敏感的生成式AI至关重要,但受限于内存、延迟和能源预算。量化可以减小模型大小和成本,但受到新兴非易失性存储器中设备噪声的影响,而传统内存层次结构进一步限制了效率。SRAM提供快速访问但密度低,DRAM必须同时容纳静态权重和动态KV缓存,这导致带宽竞争,而Flash虽然密度高,但主要用于初始化并在推理期间保持非活动状态。这些限制突显了需要为LLM推理量身定制的混合内存组织。我们提出了异常感知量化与内存协同设计(QMC),一种无需重新训练的量化方法,具有新颖的异构内存架构。QMC在SLMs中识别内群和外群权重,将内群权重存储在紧凑的多级电阻随机存取存储器(ReRAM)中,同时将关键外群权重以高精度存储在芯片上磁阻随机存取存储器(MRAM)中,从而减轻噪声引起的退化。在语言建模和推理基准测试中,QMC在使用先进算法和混合数据格式的情况下,优于并匹配了最先进的量化方法,同时在仅算法评估和现实部署设置下实现了更大的压缩。具体而言,与最新边缘AI平台上的SoTA量化方法相比,QMC在内存使用上减少了6.3x-7.3x,在外部数据传输上减少了7.6x,在能耗上减少了11.7x,在延迟上减少了12.5x,相比FP16,确立了QMC作为可扩展、部署就绪的协同设计,用于高效的设备端推理。

英文摘要:

Deploying Small Language Models (SLMs) on edge platforms is critical for real-time, privacy-sensitive generative AI, yet constrained by memory, latency, and energy budgets. Quantization reduces model size and cost but suffers from device noise in emerging non-volatile memories, while conventional memory hierarchies further limit efficiency. SRAM provides fast access but has low density, DRAM must simultaneously accommodate static weights and dynamic KV caches, which creates bandwidth contention, and Flash, although dense, is primarily used for initialization and remains inactive during inference. These limitations highlight the need for hybrid memory organizations tailored to LLM inference. We propose Outlier-aware Quantization with Memory Co-design (QMC), a retraining-free quantization with a novel heterogeneous memory architecture. QMC identifies inlier and outlier weights in SLMs, storing inlier weights in compact multi-level Resistive-RAM (ReRAM) while preserving critical outliers in high-precision on-chip Magnetoresistive-RAM (MRAM), mitigating noise-induced degradation. On language modeling and reasoning benchmarks, QMC outperforms and matches state-of-the-art quantization methods using advanced algorithms and hybrid data formats, while achieving greater compression under both algorithm-only evaluation and realistic deployment settings. Specifically, compared against SoTA quantization methods on the latest edge AI platform, QMC reduces memory usage by 6.3x-7.3x, external data transfers by 7.6x, energy by 11.7x, and latency by 12.5x when compared to FP16, establishing QMC as a scalable, deployment-ready co-design for efficient on-device inference.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑