arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2603.14591cs.LGcs.AIcs.IR
收藏

FlashHead: 语言模型推理中分类头的高效替换方案

FlashHead: Efficient Drop-In Replacement for the Classification Head in Language Model Inference

Wilhelm Tranheden, Shahnawaz Ahmed, Devdatt Dubhashi, Jonna Matthiesen, Hannes von Essen

AI总结:

FlashHead通过信息检索原理优化语言模型分类头,实现推理效率提升1.75倍,同时保持输出准确性,为更小的模型在消费级硬件上应用奠定基础。

摘要:

语言模型日益采用优化于消费设备的小型架构,在此环境下,推理效率是主要限制因素。同时,词汇量持续快速增长,使分类头成为关键瓶颈,占模型参数的60%和推理计算的50%。我们引入FlashHead,首个无需训练且硬件友好的高效分类头替换方案。FlashHead基于信息检索原理,将输出头的计算重新定义为检索问题而非对完整词汇的密集分类。FlashHead引入四个关键创新:(1)平衡聚类方案,将词汇分区结构化为紧凑的硬件高效张量;(2)扩展多探针检索至语言模型头部,使数千个聚类并行评分;(3)新颖的推理时采样机制,扩展检索至顶级token之外,实现词汇全范围的概率采样;(4)选择性量化,实现头部有效低比特计算。实验表明,FlashHead在Llama-3.2、Gemma-3和Qwen-3上实现模型级推理加速高达1.75倍,同时保持输出准确性。通过克服分类头瓶颈,FlashHead建立高效推理新基准,移除开发更小、更强大模型以适应消费级硬件的关键障碍。

英文摘要:

Language models are increasingly adopting smaller architectures optimized for consumer devices. In this setting, inference efficiency is the primary constraint. Meanwhile, vocabulary sizes continue to grow rapidly, making the classification head a critical bottleneck that accounts for up to 60\% of model parameters, and 50\% of inference compute. We introduce FlashHead, the first efficient drop-in replacement for the dense classification head that is training-free and hardware-friendly. FlashHead builds on principles from information retrieval, reframing that computation at the output head as a retrieval problem rather than a dense classification over the full vocabulary. FlashHead introduces four key innovations: (1) a balanced clustering scheme that structures vocabulary partitions into compact hardware-efficient tensors, (2) extending multiprobe retrieval to language model heads, enabling thousands of clusters to be scored in parallel, (3) a novel inference-time sampling mechanism that extends retrieval beyond top tokens, enabling probabilistic sampling across the full vocabulary, and (4) selective quantization, enabling effective low-bit computation in the head. Experiments on Llama-3.2, Gemma-3, and Qwen-3 show that FlashHead delivers model-level inference speedups of up to \textbf{1.75x} which maintaining output accuracy compared to the original head. By overcoming the classification head bottleneck, FlashHead establishes a new benchmark for efficient inference and removes a key barrier to developing smaller, capable models for consumer hardware.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑