arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2605.26778cs.AI
收藏

归因盲点:检测语言模型何时依赖记忆而非检索到的上下文

The Attribution Blind Spot: Detecting When Language Models Rely on Memory Rather Than Retrieved Context

Zhejiang University(浙江大学);Binjiang Institute of Zhejiang University(浙江大学滨江研究院);National Fintech Evaluation Center(国家金融科技评估中心);Hangzhou Dianzi University(杭州电子科技大学);Shanghai Jiao Tong University(上海交通大学)

Zhe Yu, Wenpeng Xing, Yunzhao Wei, Bo Yang, Chen Ye, Gaolei Li, Meng Han

AI总结:

本文提出计算现实监控(CRM)方法,通过比较有无上下文时的内部表征差异,检测语言模型是否依赖预训练记忆而非检索到的上下文进行生成,解决了输出级监控无法识别的归因盲点问题。

摘要:

检索增强生成承诺将语言模型输出锚定于外部证据,然而该领域缺乏可靠方法来验证检索到的上下文是否实际主导了生成——这是任何高风险部署的前提。标准假设(上下文一致的输出意味着上下文主导的输出)在检索到的文档与模型预训练数据重叠时失效:模型可以完全从参数化记忆中生成看似忠实的文本,且两种途径产生无法区分的输出。我们将此失败命名为归因盲点,并引入计算现实监控(CRM)来解决它。CRM 操作化了源自认知科学现实监控框架的一个原则:比较有上下文和无上下文时的内部表征,揭示了输出级监控系统系统性遗漏的基于成员条件的表征分歧。CRM 并不证明单个生成使用了哪个来源;它检测预训练暴露是否留下可测量的内部轨迹特征,从而为来源归因建立必要的基础。在跨越三个系列的九个模型变体中,这种分歧集中在架构特定的层模式中,得到块级噪声干预的汇聚支持,并在任务和数据集上泛化,而在领域混淆的基准上消失。归因盲点是可以测量且部分可解决的:内部表征携带输出级不可见的诊断信号,为系统建立基础,使其对证据来源的内部意识支配其外部行为。

英文摘要:

Retrieval-augmented generation promises to ground language model outputs in external evidence, yet the field has no reliable way to verify whether retrieved context actually governs generation -- a prerequisite for any high-stakes deployment. The standard assumption, that context-consistent output implies context-governed output, breaks when the retrieved document overlaps with the model's pretraining data: the model can produce faithful-looking text entirely from parametric memory, and both pathways yield indistinguishable output. We name this failure the attribution blind spot and introduce Computational Reality Monitoring (CRM) to address it. CRM operationalizes a principle adapted from cognitive science's reality monitoring framework: comparing internal representations with and without context reveals membership-conditioned representational divergence that output-level monitors systematically miss. CRM does not certify which source an individual generation used; it detects whether pretraining exposure leaves a measurable internal trajectory signature, establishing a necessary substrate for source attribution. Across nine model variants spanning three families, this divergence concentrates in architecture-specific layer patterns, receives converging support from block-level noise intervention, and generalizes across tasks and datasets while collapsing on domain-confounded benchmarks. The attribution blind spot is measurable and partially addressable: internal representations carry a diagnostic signal invisible at the output level, establishing a foundation for systems whose internal awareness of evidence provenance governs their external behavior.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑