arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2409.05448cs.CL
收藏

语言模型中绑定的表征分析

Representational Analysis of Binding in Language Models

Tohoku University(东北大学);RIKEN AIP(理化学研究所人工智能项目);MBZUAI(穆罕默德·本·扎耶德人工智能大学)

Qin Dai, Benjamin Heinzerling, Kentaro Inui

AI总结:

本文研究语言模型的实体—属性绑定机制,通过 PCA 等降维方法定位主要编码顺序信息的 Ordering ID 低秩子空间,并证明沿该方向编辑表征可因果改变绑定结果。

摘要:

实体追踪对于复杂推理至关重要。为了执行上下文内实体追踪,语言模型(LM)必须将实体与其属性绑定(例如,将容器与其内容物绑定),以便为给定实体回忆属性。例如,给定一个提到“咖啡在盒子 Z 中,石头在盒子 M 中,地图在盒子 H 中”的上下文,为了随后推断“盒子 Z 装有咖啡”,LM 必须将“盒子 Z”与“咖啡”绑定。为了解释 LM 的绑定行为,现有研究引入了一种 Binding ID 机制,并指出 LM 使用一个称为 Binding ID(BI)的抽象概念在内部标记实体—属性对。然而,他们尚未从实体激活中捕获直接决定绑定行为的 Ordering ID(OI)。在本工作中,我们通过定位 OI 并证明 OI 与绑定行为之间的因果关系,为 BI 机制提供了一种新视角。具体而言,通过利用降维方法(例如 PCA),我们发现 LM 的激活中存在一个低秩子空间,该子空间主要编码实体与属性的顺序(即 OI)。此外,我们还发现了 OI 对绑定的因果效应:当沿 OI 编码方向编辑表征时,LM 倾向于相应地将给定实体与其他属性绑定。例如,通过沿 OI 编码方向修补激活,我们可以使 LM 推断“盒子 Z 装有石头”和“盒子 Z 装有地图”。

英文摘要:

Entity tracking is essential for complex reasoning. To perform in-context entity tracking, language models (LMs) must bind an entity to its attribute (e.g., bind a container to its content) to recall attribute for a given entity. For example, given a context mentioning ``The coffee is in Box Z, the stone is in Box M, the map is in Box H'', to infer ``Box Z contains the coffee'' later, LMs must bind ``Box Z'' to ``coffee''. To explain the binding behaviour of LMs, existing research introduces a Binding ID mechanism and states that LMs use a abstract concept called Binding ID (BI) to internally mark entity-attribute pairs. However, they have not captured the Ordering ID (OI) from entity activations that directly determines the binding behaviour. In this work, we provide a novel view of the BI mechanism by localizing OI and proving the causality between OI and binding behaviour. Specifically, by leveraging dimension reduction methods (e.g., PCA), we discover that there exists a low-rank subspace in the activations of LMs, that primarily encodes the order (i.e., OI) of entity and attribute. Moreover, we also discover the causal effect of OI on binding that when editing representations along the OI encoding direction, LMs tend to bind a given entity to other attributes accordingly. For example, by patching activations along the OI encoding direction we can make the LM to infer ``Box Z contains the stone'' and ``Box Z contains the map''.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑