RECALL：语言模型中的类库行为通过自引用因果循环得到增强
RECALL: Library-Like Behavior In Language Models is Enhanced by Self-Referencing Causal Cycles
AI总结：
本文提出自引用因果循环（RECALL）机制，使LLM能绕过反转诅咒，通过循环token连接训练数据，增强从后续token回忆前文的能力，并经形式化与实验验证。
摘要：
我们引入了自引用因果循环（缩写为RECALL）的概念——这是一种使大型语言模型（LLM）能够绕过单向因果性限制的机制，而单向因果性正是被称为反转诅咒的现象的基础。当LLM被提示以顺序数据时，它常常无法回忆前面的上下文。例如，当我们要求LLM回忆美国国歌中“O say does that star-spangled banner yet wave”这一行之前的歌词时，它常常无法正确返回“Gave proof through the night that our flag was still there”——这是由于反转诅咒。它之所以发生，是因为像ChatGPT和Llama这样的语言模型基于前面的token生成文本，要求事实以一致的token顺序被学习和重现。虽然反转诅咒通常被视为一种限制，但我们提供了另一种观点的证据：它在实践中并不总是一个障碍。我们发现RECALL是由我们称之为循环token的东西驱动的——这些序列连接训练数据的不同部分，使得能够从后续token中回忆前面的token。通过严格的概率形式化和受控实验，我们证明了它们诱导的循环如何影响模型重现信息的能力。为了促进可复现性，我们在https://anonymous.4open.science/r/remember-B0B8/提供了我们的代码和实验细节。
英文摘要：
We introduce the concept of the self-referencing causal cycle (abbreviated RECALL) - a mechanism that enables large language models (LLMs) to bypass the limitations of unidirectional causality, which underlies a phenomenon known as the reversal curse. When an LLM is prompted with sequential data, it often fails to recall preceding context. For example, when we ask an LLM to recall the line preceding "O say does that star-spangled banner yet wave" in the U.S. National Anthem, it often fails to correctly return "Gave proof through the night that our flag was still there" - this is due to the reversal curse. It occurs because language models such as ChatGPT and Llama generate text based on preceding tokens, requiring facts to be learned and reproduced in a consistent token order. While the reversal curse is often viewed as a limitation, we offer evidence of an alternative view: it is not always an obstacle in practice. We find that RECALL is driven by what we designate as cycle tokens - sequences that connect different parts of the training data, enabling recall of preceding tokens from succeeding ones. Through rigorous probabilistic formalization and controlled experiments, we demonstrate how the cycles they induce influence a model's ability to reproduce information. To facilitate reproducibility, we provide our code and experimental details at https://anonymous.4open.science/r/remember-B0B8/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
