发表机构
Utrecht University; Queen Mary University of London（乌得勒支大学; 伦敦玛丽女王大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
本研究结合机制可解释性与注意力分析，探究LLMs复数指代的单复数实体表征与检索机制，发现负责共指信息处理的注意力头及与人类一致的复数代词偏好规律。
AI 中文摘要
共指消解是上下文推理中的一项重要任务。本文研究用于复数指代的单复数实体表征与检索机制，结合机制可解释性与注意力模式分析，探究大型语言模型（LLMs）预测代词回指前文实体的过程。通过一系列因果干预技术，发现一组注意力头负责：（1）在输入中表征共指信息；（2）识别构成复数指代的实体；（3）将信息传递给负责选择先行词并预测代词的组件。还发现LLMs在复数代词偏好上与人类一致，具体而言，复数结构中的实体若在本体论上相似且由连词“and”连接，则更可能被作为复数实体指代。
英文摘要
Coreference resolution is an important task in contextual reasoning. In this paper, we investigate the mechanism for representing and retrieving singular and plural entities for plural reference. We use a combination of mechanistic interpretability and attention pattern analysis to study the process in which LLMs predict a pronoun to refer back to previously mentioned entities. Using a range of causal intervention techniques, we find a set of attention heads that are responsible for (1) representing coreference information in the input, (2) identifying entities that form a plural reference, (3) transferring the information to the component that is responsible for selecting the antecedents and predicting the pronoun. We also find that LLMs align with humans in preference for plural pronoun. Specifically, entities in a plural construction are more likely to be referred to as a plural entity if they are ontologically similar and are linked by the conjunction "and".
