因果干预揭示用于常识语言理解的隐式情境模型
Causal interventions expose implicit situation models for commonsense language understanding
AI总结：
本文通过对 transformer 模型进行因果干预，识别 WSC 中传播上下文线索并指导代词消解的小型注意力头回路，并借句法对照揭示隐式情境模型的独特构建通路。
摘要：
长期以来，关于人类语言处理的解释都诉诸隐式的“情境模型”，这些模型用相关但未明说的世界知识来丰富理解。本文将因果干预技术应用于近期的 transformer 模型，以分析其在 Winograd Schema Challenge（WSC）上的表现；在该挑战中，单个上下文线索会改变对一个歧义代词的解释。我们识别出一个由注意力头组成的相对较小的回路，它负责传播来自上下文词的信息，从而引导代词最终关注候选名词短语中的哪一个。随后，我们比较该回路在一个紧密匹配的“句法”控制条件中的行为；在该控制条件下，情境模型并非严格必要。这些分析表明，隐式情境模型可通过不同的通路构建，以指导代词消解。
英文摘要：
Accounts of human language processing have long appealed to implicit ``situation models'' that enrich comprehension with relevant but unstated world knowledge. Here, we apply causal intervention techniques to recent transformer models to analyze performance on the Winograd Schema Challenge (WSC), where a single context cue shifts interpretation of an ambiguous pronoun. We identify a relatively small circuit of attention heads that are responsible for propagating information from the context word that guides which of the candidate noun phrases the pronoun ultimately attends to. We then compare how this circuit behaves in a closely matched ``syntactic'' control where the situation model is not strictly necessary. These analyses suggest distinct pathways through which implicit situation models are constructed to guide pronoun resolution.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
