arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2604.21026cs.LG
收藏

MCAP:面向内存受限LLM推理的部署时层性能分析

MCAP: Deployment-Time Layer Profiling for Memory-Constrained LLM Inference

Genovation Technological Solutions Pvt Ltd(基因创新技术解决方案私人有限公司)

Anurita Das

AI总结:

MCAP通过部署时的层重要性估计,实现动态精度和内存分配决策,提升解码吞吐量并拓展内存受限下的模型运行范围。

摘要:

在异构硬件上部署大语言模型通常受限于内存而非计算能力。我们引入MCAP(蒙特卡洛激活分析),一种部署时的逐层重要性估计器,能够在目标设备上实现动态精度和内存放置决策。MCAP生成轻量级的逐层信号,驱动精度调度(W4A8 vs. W4A16)和驻留层级(GPU、RAM、SSD),使同一组权重能够在不同内存预算下运行。我们的系统NVE在NVIDIA T4上实现比llama-cpp Q4_0高1.5-1.8倍的解码吞吐量,并使模型能够在无需修改权重的情况下在之前不可行的内存范围内运行。

英文摘要:

Deploying large language models to heterogeneous hardware is often constrained by memory, not compute. We introduce MCAP (Monte Carlo Activation Profiling), a load-time per-layer importance estimator that enables dynamic precision and memory placement decisions on the target device. MCAP produces a lightweight per-layer signal that drives both precision dispatch (W4A8 vs. W4A16) and residency tier (GPU, RAM, SSD), allowing a single set of weights to operate across diverse memory budgets. Our system, NVE, achieves 1.5-1.8x higher decode throughput than llama-cpp Q4_0 on NVIDIA T4 and enables models to run in memory regimes previously infeasible without modifying weights.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑