发表机构
HPI / University of Potsdam; African Institute for Mathematical Sciences; Department of Computer Science Aalborg University; Department of Computer Science University of Cape Town; Polytechnique Montréal（波茨坦大学哈索·普拉特纳数字工程学院; 非洲数学科学研究所; 奥尔堡大学计算机科学系; 开普敦大学计算机科学系; 蒙特利尔理工大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
研究修改预训练信号能否使语言模型从参数化记忆转向基于证据的推理，引入无知识语言模型（KLLMs），其在命名实体匿名化语料库上预训练，实验表明能降低闭卷事实记忆，在多任务基准上表现优，提升鲁棒性与校准，为语言模型训练提供新思路。
AI 中文摘要
语言模型在其参数中编码了大量事实知识，当这些知识过时、不完整或与所提供的上下文不一致时，可能导致不可靠的行为。在这项工作中，我们研究修改预训练信号是否能系统地使模型从参数化记忆转向基于证据的推理。我们引入了无知识语言模型（KLLMs），这是一种针对语言模型的根本不同的认知训练范式，在命名实体被匿名化的语料库上进行预训练，从而消除了实体链接事实监督的主要渠道。这种干预大幅降低了闭卷事实记忆，同时通常提高了在将相关信息作为上下文提供的任务上的性能。在多个模型规模上，KLLMs在上下文问答、事实核查和幻觉检测基准上始终优于匹配的基线。至关重要的是，在证据不完美的基于检索的设置中，KLLMs表现出更高的鲁棒性，相对于标准语言模型实现了高达20%-25%的相对增益。它们还表现出更好的校准，具有改进的ECE、Brier分数和AUROC，以及更可靠的弃权行为。我们的结果表明，在预训练期间抑制实体链接监督会导致认知行为的转变：KLLMs较少依赖参数化知识，更多地依赖外部证据，从而在现实条件下提高了可靠性。这表明，通过提供一个对证据更敏感的基础模型，预训练时对知识获取的控制可以补充检索增强和基于工具的系统。
英文摘要
Language models encode substantial factual knowledge in their parameters, which can lead to unreliable behavior when this knowledge is outdated, incomplete, or misaligned with the provided context. In this work, we study whether modifying the pretraining signal can systematically shift models away from parametric recall and toward evidence-grounded reasoning. We introduce Knowledge--''Less'' Language Models (KLLMs), a fundamentally different epistemic training paradigm for LLMs, which are pretrained on corpora in which named entities are anonymized, thereby removing a primary channel for entity-linked factual supervision. This intervention substantially reduces closed-book factual recall, while often improving performance on tasks where relevant information is provided as context. Across multiple model scales, KLLMs consistently outperform matched baselines on contextual question answering, fact verification, and hallucination detection benchmarks. Crucially, in retrieval-grounded settings with imperfect evidence, KLLMs show improved robustness and achieve up to 20--25\% relative gains over standard language models. They further exhibit better calibration, with improved ECE, Brier score, and AUROC, as well as more reliable abstention behavior. Our results demonstrate that suppressing entity-linked supervision during pretraining induces a shift in epistemic behavior: KLLMs rely less on parametric knowledge and more on external evidence, leading to improved reliability under realistic conditions. This suggests that pretraining-time control over knowledge acquisition can complement retrieval-augmented and tool-based systems by providing a more evidence-sensitive base model.
