GRINQH：基于分级输入的量化层次结构用于高效LLM生成
GRINQH: Graded Input-based Quantization Hierarchy for Efficient LLM Generation
AI总结：
提出GRINQH框架，利用激活幅度动态分配权重通道精度，统一量化与稀疏化，加速LLM解码阶段，在Llama3和Qwen3上优于现有方法，实现2位生成。
摘要：
自回归解码与LLM主要受限于GPU内存带宽，尤其是在边缘计算环境中。虽然量化对于缓解这一瓶颈至关重要，但大多数现有方法将推理视为统一过程，未能考虑计算受限的预填充阶段与内存受限的解码阶段之间的不对称性。我们提出GRINQH（基于分级输入的量化层次结构），一种仅权重的训练后量化框架，通过统一量化与稀疏化来加速解码。GRINQH利用激活幅度作为计算重要性的代理，动态地将权重通道分配到不同精度级别，从而在解码过程中实现灵活的平均位宽。在Llama3和Qwen3模型上的评估表明，GRINQH在可比的3位和4位设置下优于最先进的固定精度和混合精度基线，甚至实现了有效的2位生成。我们通过利用自定义GPU内核中的分层嵌套内存布局进行多精度存储，实验验证了理论加速。最终，GRINQH为LLM生成建立了新的最先进帕累托前沿，实现了生成质量与推理速度之间的动态权衡。
英文摘要：
Autoregressive decoding with LLMs is primarily bottlenecked by GPU memory bandwidth, especially in edge-computing settings. While quantization is essential for mitigating this bottleneck, most existing methods treat inference as a uniform process and fail to account for the asymmetry between the compute-bound prefill stage and the memory-bound decoding stage. We propose GRINQH (GRaded INput-based Quantization Hierarchy), a weight-only post-training quantization framework that accelerates decoding by unifying quantization and sparsification. GRINQH leverages activation magnitudes as a proxy for computational importance to dynamically assign weight channels to different precision levels, enabling flexible average bit widths during decoding. Evaluated on Llama3 and Qwen3 models, GRINQH outperforms state-of-the-art fixed- and mixed-precision baselines at comparable 3- and 4-bit settings, even enabling effective 2-bit generation. We experimentally verify theoretical speedups by leveraging a hierarchical nested memory layout for multi-precision storage in a custom GPU kernel. Ultimately, GRINQH establishes a new state-of-the-art Pareto frontier for LLM generation, enabling a dynamic trade-off between generation quality and inference speed.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
