对大型语言模型中代词忠实性的机制理解
A Mechanistic Understanding of Pronoun Fidelity in LLMs
AI总结：
通过因果分析发现，代词忠实性由组实体绑定、近因偏差和刻板印象偏差三种因果子空间共同作用，解释了91-99.5%的行为。
摘要：
忠实且稳健的代词使用对于公平和连贯的生成至关重要，然而当多个指代对象使用不同代词时，大型语言模型大多会失败。为了研究推理、重复和偏差在此任务中的相互作用，先前的工作完全依赖行为方法，这可能无法反映模型的内部运作。因此，我们提供了关于代词忠实性的机制性、模型内部视角，测试了三种机制——组实体绑定（G）、近因偏差（R）和刻板印象偏差（S）——是否在多个SOTA语言模型中因果实现。使用无界分布式对齐搜索，我们发现三者作为因果子空间共存，分布在网络深度上。没有单一机制能完全解释模型行为，但三者的组合一致地解释了91-99.5%。注意力头分析进一步揭示了两种竞争的复制路径；组绑定和刻板印象共享一个局部化的概念级路径，检索绑定的职业-代词单元，而近因使用分布式的令牌级路径，重复表面形式。总之，代词忠实性源于同时活跃的因果子空间之间的竞争。
英文摘要：
Faithful and robust pronoun use is important for fair and coherent generations, yet large language models largely fail when multiple referents use different pronouns. To study the interplay of reasoning, repetition, and bias in this task, prior work relies exclusively on behavioural approaches, which may not reflect a model's internal workings. Therefore, we provide a mechanistic, model-internal perspective on pronoun fidelity, testing whether three mechanisms -- group entity binding (G), recency bias (R), and stereotypical bias (S) -- are causally implemented across several SOTA language models. Using Boundless Distributed Alignment Search, we find all three coexist as causal subspaces distributed across network depth. No single mechanism fully explains model behaviour, but a combination of the three consistently accounts for 91-99.5%. An attention head analysis further reveals two competing copying routes; group binding and stereotype share a localized concept-level route that retrieves a bound occupation-pronoun unit, while recency uses a distributed token-level route that repeats surface forms. In sum, pronoun fidelity arises from competition between simultaneously active causal subspaces.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
