上下文为王：上下文规范如何塑造概念的几何结构

中文摘要 AI 辅助

大型语言模型将结构化概念置于几何上忠实的流形上：工作日位于一个圆上，月份位于另一个圆上，通常被视为网络存储和查找的固定世界模型。我们表明上下文为王：模型实际使用的结构由上下文中的规范设置。一个声明性规则不仅确定几何编码的关系，还确定其拓扑类型：相同的令牌可以根据命令形成循环或分支树，甚至基于任意的、无意义的令牌构建，无需继承先验知识，而重新标记的存储形状则无法做到。当规范与强大的预训练先验冲突时，在有能力的模型中，上下文设置的几何结构会主导它，从相同的激活中读取（与强加结构的表征相似性为0.6 - 0.9，与先验的相似性接近零），涵盖我们测试的先验和我们研究的两个模型家族（Gemma、Qwen）。激活修补表明该映射被因果性地使用，而不是探测相关性：将一个实体的激活与另一个实体的激活交换会使模型在强加顺序下用另一个实体的后继者回答。即使在小型和基础模型中也很容易形成粗略的映射；决定能否清晰使用它的是模型规模：清晰的主导和因果交叉仅出现在较大的模型中（高达Gemma - 31B和Qwen - 27B），在较小的模型中会减弱或反转，所以同一模型家族中较大模型存在的机制在较小模型中可能不存在。我们不确定模型是重新构建这个几何结构还是重新配置存储的结构；从操作上讲，它使用的几何结构是上下文指定的那个。

英文摘要

Large language models place structured concepts on geometrically faithful manifolds: weekdays lie on a circle, months on another, usually taken to be a fixed world-model the network stores and looks up. We show that context is king: the structure a model actually uses is set by the in-context specification. A declarative rule fixes not only which relations the geometry encodes but its topology type: the same tokens form a cycle or a branching tree on command, built even on arbitrary, meaning-free tokens with no prior to inherit, which a relabeled stored shape cannot do. When the specification conflicts with a strong pretrained prior, the context-set geometry dominates it in capable models, read from the same activations (representational similarity 0.6--0.9 to the imposed structure versus near-zero to the prior), across the priors we test and both families we study (Gemma, Qwen). Activation patching shows the map is causally used, not a probe correlate: swapping one entity's activation for another's makes the model answer with the other entity's successor under the imposed order. A rough map forms readily, present even in small and base models; what scale gates is using it cleanly: clean dominance and the causal crossover emerge only in the larger models (up to Gemma-31B and Qwen-27B) and weaken or reverse below, so a mechanism present in a large model can be absent in a smaller one of the same family. Whether the model builds this geometry anew or reconfigures a stored one we leave open; operationally, the geometry it uses is the one the context specifies.

发表机构

• Zenity

机构由 AI 辅助整理，请以论文原文为准。