arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2601.22795cs.CL
收藏

稀疏还是密集?一种对基于Transformer的LLM计算密度的机理估计

Sparse or Dense? A Mechanistic Estimation of Computation Density in Transformer-based LLMs

Corentin Kervadec, Iuliia Lysova, Marco Baroni, Gemma Boleda

AI总结:

本文通过机理可解释性方法系统量化了基于Transformer的LLM的计算密度,发现LLM通常涉及密集计算,计算密度动态变化,且不同模型对相同输入的密度相关性显著。

摘要:

基于Transformer的大型语言模型(LLM)由数十亿参数组成,这些参数排列在深度和宽度都很大的计算图中。几项关于LLM效率优化的研究认为,可以显著剪枝参数,同时对性能影响微乎其微。这表明计算并非均匀分布在参数上。我们引入了一种技术,系统地量化LLM中的计算密度。具体来说,我们设计了一个基于机理可解释性的密度估计器。我们实验性地测试了我们的估计器,并发现:(1)与以往假设相反,LLM处理通常涉及密集计算;(2)计算密度是动态的,即模型根据输入在稀疏和密集处理模式之间切换;(3)每个输入的密度在不同LLM之间显著相关,表明相同的输入会触发低或高密度。调查影响密度的因素,我们发现预测稀有词需要更高的密度,而增加上下文长度通常会降低密度。我们相信,我们的计算密度估计器将有助于更好地理解LLM的处理机制,挑战其符号解释。

英文摘要:

Transformer-based large language models (LLMs) are comprised of billions of parameters arranged in deep and wide computational graphs. Several studies on LLM efficiency optimization argue that it is possible to prune a significant portion of the parameters, while only marginally impacting performance. This suggests that the computation is not uniformly distributed across the parameters. We introduce here a technique to systematically quantify computation density in LLMs. In particular, we design a density estimator drawing on mechanistic interpretability. We experimentally test our estimator and find that: (1) contrary to what has been often assumed, LLM processing generally involves dense computation; (2) computation density is dynamic, in the sense that models shift between sparse and dense processing regimes depending on the input; (3) per-input density is significantly correlated across LLMs, suggesting that the same inputs trigger either low or high density. Investigating the factors influencing density, we observe that predicting rarer tokens requires higher density, and increasing context length often decreases the density. We believe that our computation density estimator will contribute to a better understanding of the processing at work in LLMs, challenging their symbolic interpretation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑