词汇参考框架:用原生语义解释语言模型的残差流
Lexical Reference Frames: Interpreting LM Residual Streams in Native Semantics
AI总结:
本文提出词汇参考框架,通过模型剩余层读取状态近似原生参考框架,其一阶框架LexFrame[1]相干性高、读取误差小,可提升归因忠实度、引导预测并支持早期退出,让模型隐藏状态具备可读性、可归因性与可引导性。
摘要:
语言模型的词汇是其原生语义:嵌入层和非嵌入层分别确定残差流两端状态的含义。在这两者之间,现有方法要么复用固定的透镜,要么将状态视为点云,均未以模型自身的方式读取状态。本文提出,每个隐藏状态都有一个原生参考框架,该框架由后续读取它的层诱导并自适应于上下文。词汇参考框架(Lexical Reference Frames)通过以下方式近似该框架:用模型自身剩余层读取状态,在位置间设置指定轮数的注意力,其余部分采用典型注意力输出,且不拟合模型的输出。实验结果显示,固定方向的透镜无法超过原生框架的相干性平方根,SmolLM3-3B和OLMo3-7B模型中层的该值分别为0.30和0.35;而一阶框架LexFrame[1]的相干性达到0.97和0.97,读取状态与模型输出的差异在0.16 nat以内。此外,LexFrame[1]的系数归因比归因修补更忠实,其词汇基元可在不影响其他token的情况下引导预测,且其置信度读数可让早期退出跳过0.45和0.42的层,而透镜几乎无法跳过任何层。综上,词汇参考框架使模型的每个隐藏状态都具备可读性、可归因性和可引导性。
英文摘要:
The vocabulary of a language model is its native semantics, and the embedding and unembedding fix the meaning of a state at the two ends of the residual stream. Between them, existing methods reuse a fixed lens or treat states as a point cloud, and neither reads a state as the model does. We argue that every hidden state has a native reference frame, induced by the layers that remain and adaptive to the context. Lexical reference frames approximate it in a model-native and context-aware way, reading a state through the remaining layers with a given number of rounds of attention between positions and a typical attention output for the rest, fitting nothing to the model's output. No lens with fixed directions per layer can reach a mean cosine with the native frame's lexical readouts above the square root of their coherence, 0.30 and 0.35 at mid-depth of SmolLM3-3B and OLMo3-7B. LexFrame[1], of order 1, reaches 0.97 and 0.97, and its readings are on average within 0.16 and 0.15 nats of the model's output. Its coefficient differences attribute predictions more faithfully than attribution patching, and its lexical basis steers predictions while sparing other tokens. On average, its readings settle on the model's top token over the final 0.82 and 0.81 of the layers, those of lenses over at most 0.11. These frames thus make every hidden state readable, attributable and steerable.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。