arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2610.12939cs.CLcs.AI
收藏

激活波探索:分析用于推测解码的大语言模型隐藏状态的信息与几何特性

Surfing the Activation Wave: Analyzing the Information and Geometry of LLM Hidden States for Speculative Decoding

Qualcomm AI Research(高通人工智能研究院);Stanford University(斯坦福大学)

Raghavv Goel, Sudhanshu Agrawal, Mukul Gagrani, Risheek Garrepalli, Mingu Lee

AI总结:

本研究提出基于信息性与多样性准则的多层隐藏状态选择框架,结合熵与CKA分析,在EAGLE3式推测解码中优于固定基线选择,为相关应用奠定基础。

摘要:

大语言模型(LLM)不同层的解码器隐藏状态通常编码互补信息,但现有推测解码方法通常依赖基于固定位置启发式选择的少量目标模型层。最优层子集的选择本质上是组合问题,即使对于中等深度的模型,穷举评估也不可行。本研究提出一种用于推测解码的多层隐藏状态选择的原则性框架,基于两个互补准则:通过矩阵熵分析衡量的信息性,以及用中心核对齐(CKA)量化的多样性。我们研究了逐层香农熵、隐藏状态拼接下的聚合熵,以及跨深度的成对CKA动态,揭示了解码器表示中不同的几何与信息分区。基于这些发现,我们提出基于熵和CKA的层选择算法,倾向于选择既高信息性又低冗余的隐藏状态集。作为该框架的一部分,我们引入了 token-熵过滤,它移除与低预测熵 token 相关的隐藏状态,从而提升层选择统计的质量。在EAGLE3式推测解码中,我们选择的集优于固定基线选择,包括默认的[2, N//2, N-3]规则,同时能更清晰地解释为何部分默认规则仍能保持竞争力。我们的结果为推测解码的多层隐藏状态选择提供了实用方案,为LLM解码器状态的几何与信息特性提供了见解,并为将类似分析扩展至相关应用奠定了基础。

英文摘要:

Decoder hidden states from different layers of large language models (LLMs) often encode complementary information. However, existing speculative decoding methods typically rely on a small set of target-model layers chosen using fixed, position-based heuristics. Selecting an optimal subset of layers is inherently a combinatorial problem, making exhaustive evaluation infeasible even for moderately deep models. In this work, we present a principled framework for multi-layer hidden-state selection for speculative decoding based on two complementary criteria: informativeness, measured through matrix-entropy analysis, and diversity, quantified using centered kernel alignment (CKA). We investigate layer-wise Shannon entropy, aggregate entropy under hidden-state concatenation, and pairwise CKA dynamics across depth, revealing distinct geometric and informational compartments within decoder representations. Motivated by these findings, we propose entropy and CKA-based layer selection algorithms that favor hidden-state sets that are both highly informative and minimally redundant. As part of this framework, we introduce token-entropy filtering, which removes hidden states associated with low predictive-entropy tokens, thereby improving the quality of layer-selection statistics. In EAGLE3-style speculative decoding, our selected sets outperform fixed baseline choices, including the default $[2, N//2, N-3]$ rule, while providing a clearer explanation of why some defaults can remain competitive. Our results provide a practical recipe for multi-layer hidden-state selection for speculative decoding, insights into the geometry and information in LLM decoder states, and a foundation for extending similar analysis to related applications.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑