AI 中文总结
本研究通过干预实验揭示混合语言模型中两跳检索的关键交接机制：注意力层转换关键，循环层携带并存储关键，后续层可按关键寻址查询记忆。
AI 中文摘要
一个两跳问题使语言模型检索两次：一次用于生成桥梁实体，一次用于使用该实体进行检索。Transformer在其早期层中解析该实体。混合模型将大部分注意力替换为循环状态，因此关键在何处变得可用，以及在何处用于回答，尚不清楚。从“球属于Alice”和“Alice在花园中”回答“球在哪里？”依赖于问题未提及的名字Alice。模型可以通过Alice（它计算出的关键）或通过事实在提示中的位置到达花园。在十二个模型（密集和混合）中，我们将一个隐藏状态从一个故事移动到另一个故事，其中两条路线通向不同位置，并读取模型给出的位置。在我们测试的每个模型中，注意力层转换关键，跨越该层会消除其可用效果，全部在没有注意力跟随的地方。在顺序混合模型中，这使得答案成为交接：循环层向前携带关键，注意力层花费它。检索并不总是止步于此。将不同事实写入跟随最后一个注意力层的循环层的记忆中，可以将答案移向该事实，将其几率乘以1.3到2.7倍，而哪些混合模型会这样做并不由其架构或训练决定。该读取由关键寻址，而非位置：混合模型中的循环状态不仅是载体，而且是后续层可以查询的记忆。
英文摘要
A two-hop question makes a language model retrieve twice: once to produce a bridge entity, and once to retrieve with it. Transformers resolve that entity in their early layers. Hybrid models replace most of the attention with a recurrent state, so where the key becomes usable, and where it is spent on an answer, is not known. Answering "Where is the ball?" from "the ball belongs to Alice" and "Alice is in the garden" turns on Alice, a name the question does not mention. A model could reach garden through Alice, the key it computed, or through where the fact sits in the prompt. Across twelve models, dense and hybrid, we move a hidden state from one story into another where the two routes lead to different places, and read off which place the model gives. An attention layer converts the key in every model we tested, and crossing that layer removes its usable effect, all of it where no attention follows. In sequential hybrids this makes the answer a handoff: recurrent layers carry the key forward, and attention spends it. Retrieval does not always stop there. Writing a different fact into the memory of the recurrent layers that follow the last attention layer can move the answer toward that fact, multiplying its odds by 1.3 to 2.7, and which hybrids do this is not settled by their architecture or training. That read is addressed by the key, not by position: recurrent state in a hybrid is not only a carrier, but a memory that later layers can query.
Comments36 pages, 5 figures in the main text
