连接线索：LLM能够从离散的训练数据中推断并表达潜在结构
Connecting the Dots: LLMs can Infer and Verbalize Latent Structure from Disparate Training Data
AI总结：
本文研究了LLM通过归纳性上下文外推理（OOCR）从离散训练数据中推断潜在结构并应用于下游任务的能力，发现前沿LLM能实现此推理但可靠性有限，这为LLM知识监控带来挑战。
摘要：
应对大型语言模型（LLM）安全风险的一种方法是从其训练数据中审查危险知识。虽然这移除了显式信息，但隐式信息可能仍散布于各种训练文档中。LLM能否通过拼凑这些隐式线索来推断出被审查的知识？作为回答该问题的一步，我们研究了归纳性上下文外推理（OOCR），这是一种泛化能力，LLM据此从分布于训练文档中的证据推断潜在信息，并在无上下文学习的情况下将其应用于下游任务。通过包含五个任务的套件，我们证明了前沿LLM能够执行归纳性OOCR。在一个实验中，我们在仅包含一个未知城市与其他已知城市之间距离的语料库上微调了一个LLM。值得注意的是，在没有上下文示例或思维链的情况下，该LLM能够表达出未知城市是巴黎，并利用这一事实回答下游问题。进一步的实验表明，仅在单独的抛硬币结果上训练的LLM能够表达硬币是否有偏差，而仅在$(x,f(x))$对上训练的LLM能阐述$f$的定义并计算逆函数。尽管OOCR在多种情况下取得成功，我们也表明它是不可靠的，特别是对于学习复杂结构的较小LLM。总体而言，LLM在无显式上下文学习下“连接线索”的能力，对监控和控制LLM所获取的知识构成了潜在障碍。
英文摘要：
One way to address safety risks from large language models (LLMs) is to censor dangerous knowledge from their training data. While this removes the explicit information, implicit information can remain scattered across various training documents. Could an LLM infer the censored knowledge by piecing together these implicit hints? As a step towards answering this question, we study inductive out-of-context reasoning (OOCR), a type of generalization in which LLMs infer latent information from evidence distributed across training documents and apply it to downstream tasks without in-context learning. Using a suite of five tasks, we demonstrate that frontier LLMs can perform inductive OOCR. In one experiment we finetune an LLM on a corpus consisting only of distances between an unknown city and other known cities. Remarkably, without in-context examples or Chain of Thought, the LLM can verbalize that the unknown city is Paris and use this fact to answer downstream questions. Further experiments show that LLMs trained only on individual coin flip outcomes can verbalize whether the coin is biased, and those trained only on pairs $(x,f(x))$ can articulate a definition of $f$ and compute inverses. While OOCR succeeds in a range of cases, we also show that it is unreliable, particularly for smaller LLMs learning complex structures. Overall, the ability of LLMs to "connect the dots" without explicit in-context learning poses a potential obstacle to monitoring and controlling the knowledge acquired by LLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
