发表机构
Tencent; Beihang University（腾讯; 北京航空航天大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
本研究通过配对探针和干预实验，揭示混合语言模型中归纳电路的组织机制，发现携带前驱信息集中于高效层，匹配集中于全局层，且值前令牌是关键，连接架构、电路发展与召回。
AI 中文摘要
混合语言模型可以提升能力与效率，这引出了架构互补性如何转化为习得计算的问题。我们考察了归纳的既定角色：携带前驱信息、按内容匹配源、以及复制其值。这些位置敏感且基于内容的计算如何在异构层间分配？我们引入了与层类型无关的配对探针，通过共同的块更新接口跟踪携带和匹配。在循环-全局和局部-全局混合模型中，携带集中在高效层，而匹配集中在全局接收层。测得的局部贡献集中在滞后一：即历史值之前的那个令牌。通过滞后一掩蔽、卷积移除或早期学习率降低来改变前驱支持，可以将携带和匹配在阶段间重新定位。源键恢复和固定值选择追踪了接收层对准备好的源的依赖。这些干预也改变了自然文本的召回，结果取决于配置和目标。改变局部窗口和归纳增强的训练文本改变了功能性携带和匹配的早期发展，将架构先验和训练证据与形成时间联系起来。总之，探针和干预将解释焦点上移：匹配的组织遵循携带的学习方式。值之前的令牌为混合架构、电路发展和召回之间提供了具体联系。代码可在该 https URL 获取。
英文摘要
Hybrid language models can improve capability as well as efficiency, raising the question of how architectural complementarity becomes learned computation. We examine the established induction roles of Carrying predecessor information, Matching a source by content, and Copying its value. How are these position-sensitive and content-based computations allocated across heterogeneous layers? We introduce layer-type-agnostic paired probes that track Carrying and Matching through a common block-update interface. In recurrent--global and local--global hybrids, Carrying concentrates in efficient layers and Matching in global receivers. The measured local contribution concentrates on lag one: the token immediately before the historical value. Changing predecessor support through lag-one masking, convolution removal, or early learning-rate reduction can relocate Carrying and Matching between stages. Source-key restoration and fixed-value selection trace the receiver's dependence on the prepared source. These interventions also change natural-text recall, with outcomes depending on configuration and target. Varying local windows and induction-enriched training text changes the early development of functional Carrying and Matching, connecting architectural priors and training evidence to formation timing. Together, the probes and interventions shift the explanatory focus upstream: the organization of Matching follows how Carrying is learned. The token before the value provides a concrete link between a hybrid's architecture, circuit development, and recall. Code is available in https://github.com/ckpassenger/bind-match-copy/tree/main.
Comments30 pages, including references and appendices
