在异构硬件上部署大语言模型通常受限于内存而非计算能力。我们引入MCAP(蒙特卡洛激活分析),一种部署时的逐层重要性估计器,能够在目标设备上实现动态精度和内存放置决策。MCAP生成轻量级的逐层信号,驱动精度调度(W4A8 vs. W4A16)和驻留层级(GPU、RAM、SSD),使同一组权重能够在不同内存预算下运行。我们的系统NVE在NVIDIA T4上实现比llama-cpp Q4_0高1.5-1.8倍的解码吞吐量,并使模型能够在无需修改权重的情况下在之前不可行的内存范围内运行。
英文摘要:
Deploying large language models to heterogeneous hardware is often constrained by memory, not compute. We introduce MCAP (Monte Carlo Activation Profiling), a load-time per-layer importance estimator that enables dynamic precision and memory placement decisions on the target device. MCAP produces a lightweight per-layer signal that drives both precision dispatch (W4A8 vs. W4A16) and residency tier (GPU, RAM, SSD), allowing a single set of weights to operate across diverse memory budgets. Our system, NVE, achieves 1.5-1.8x higher decode throughput than llama-cpp Q4_0 on NVIDIA T4 and enables models to run in memory regimes previously infeasible without modifying weights.
Sustainable LLM Inference for Edge AI: Evaluating Quantized LLMs for Energy Efficiency, Output Accuracy, and Inference Latency
SINTEF(辛泰夫(挪威研究机构));Singapore Management University(新加坡管理大学);Oslo Metropolitan University(奥斯陆城市大学);Kristiania University of Applied Sciences(克里斯蒂安尼亚应用科技大学)
本研究在Raspberry Pi 4上评估了28个量化LLM在能效、推理性能和准确率上的权衡,通过硬件级能耗分析与多数据集基准测试,为资源受限环境下的可持续LLM部署提供了可操作的见解。
Alloc-MoE: Budget-Aware Expert Activation Allocation for Efficient Mixture-of-Experts Inference
National Key Laboratory of Parallel and Distributed Computing, College of Computer Science and Technology, National University of Defense Technology(国防科技大学并行与分布计算国家重点实验室,计算机科学与技术学院)