TTQ:激活感知的测试时间量化以加速LLM的实时推理
TTQ: Activation-Aware Test-Time Quantization to Accelerate LLM Inference On The Fly
AI总结:
本文提出TTQ框架,在推理时动态压缩大模型,通过高效在线校准实现无需重训练的激活感知量化,提升推理速度并适应不同下游任务。
摘要:
为应对大基础模型巨大的计算需求,已引入无需重训练的激活感知压缩技术。然而,由于这些方法高度依赖校准数据,可能在未见的下游任务中出现领域偏移问题。我们提出一种测试时间量化(TTQ)框架,在推理时动态压缩大模型以解决此问题。通过高效的在线校准,可以即时实现激活感知量化,适应每一个提示,而无需考虑下游任务,同时实现推理加速。几项实验表明,TTQ在量化性能上优于现有最先进基线。
英文摘要:
To tackle the huge computational demand of large foundation models, activation-aware compression techniques without retraining have been introduced. However, since these methods highly rely on calibration data, domain shift issues may arise for unseen downstream tasks. We propose a test-time quantization (TTQ) framework which compresses large models on the fly at inference time to resolve this issue. With an efficient online calibration, instant activation-aware quantization can adapt every prompt regardless of the downstream tasks, yet achieving inference speedup. Several experiments demonstrate that TTQ can improve the quantization performance over state-of-the-art baselines.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。