发表机构
Cardiff University; Queen Mary University of London（卡迪夫大学; 伦敦大学女王学院）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
针对检索增强生成中证据来源可靠性问题，提出公开知识库MEDIAREF，支持可复现的低成本媒体背景核查生成，评估多种大语言模型并证明其有效性。
AI 中文摘要
基于LLM的检索增强生成（RAG）越来越多地用于自动事实核查（AFC）及相关任务。通过将LLM输出锚定在检索到的证据上，基于RAG的系统提供透明的理由，同时允许外部信息独立于底层模型进行更新。然而，现有方法通常假设检索到的证据是可靠的，尽管现实世界的信息可能存在冲突、过时，并且可能来自不可靠或有偏见的来源。最近关于*源批判性推理*的工作通过媒体背景核查（MBCs）（Schlichtkrull, 2024）解决了这一挑战，该核查评估证据来源的可信度以支持下游事实验证。然而，生成MBCs依赖于昂贵的专有搜索API，限制了可重复性。为缓解这一问题，我们引入了MEDIAREF，一个公开可用的网络文档知识库，支持对200个媒体来源的MBC生成进行可重复、低成本的评估。我们描述了一种可重复的方法来构建和更新该集合，评估了广泛使用的LLMs在MBC生成任务上的表现，并通过自动化和定性评估证明MEDIAREF支持更高质量的MBC生成。
英文摘要
LLM-based retrieval-augmented generation (RAG) is increasingly used for automated fact-checking (AFC) and related tasks. By grounding LLM outputs in retrieved evidence, RAG-based systems provide transparent justifications while allowing external information to be updated independently of the underlying model. However, existing approaches often assume retrieved evidence is reliable, although real-world information may be conflicting, outdated, and can originate from unreliable or biased sources. Recent work on *source-critical reasoning* addresses this challenge through media background checks (MBCs) (Schlichtkrull, 2024), which assess the credibility of evidence sources to support downstream fact verification. However, generating MBCs relies on costly proprietary search APIs, limiting reproducibility. To mitigate this issue, we introduce MEDIAREF, a publicly available knowledge store of web-sourced documents that enables reproducible, low-cost evaluation of MBC generation across 200 media sources. We describe a reproducible methodology for constructing and updating the collection, assess widely used LLMs on the MBC generation task, and demonstrate that MEDIAREF supports higher-quality MBC generation through both automatic and qualitative evaluation.
CommentsCode and Data: https://github.com/nedjmaou/mediaref
