解决自动事实核查中的冲突证据：关于检索增强大语言模型的研究
Resolving Conflicting Evidence in Automated Fact-Checking: A Study on Retrieval-Augmented LLMs
AI总结：
本研究首次系统评估了检索增强大语言模型在冲突证据下的事实核查性能，构建了CONFACT数据集，并证明整合来源可信度能显著提升模型解决冲突和核查事实的能力。
摘要：
通过检索机制增强的大语言模型（LLMs）在事实核查任务中通过整合外部知识展现了显著潜力。然而，当面对来自不同可信度来源的冲突证据时，其可靠性会下降。本文首次对检索增强生成（RAG）模型在存在冲突证据的情况下进行事实核查进行了系统评估。为支持本研究，我们引入了CONFACT（冲突证据用于事实核查）（数据集可在https://github.com/zoeyyes/CONFACT获取），这是一个新颖的数据集，包含与来自不同来源的冲突信息配对的问题。大量实验揭示了最先进的RAG方法的关键脆弱性，特别是在解决因媒体来源可信度差异而产生的冲突方面。为应对这些挑战，我们研究了将媒体背景信息整合到检索和生成阶段的策略。我们的结果表明，有效整合来源可信度显著增强了RAG模型解决冲突证据和提升事实核查性能的能力。
英文摘要：
Large Language Models (LLMs) augmented with retrieval mechanisms have demonstrated significant potential in fact-checking tasks by integrating external knowledge. However, their reliability decreases when confronted with conflicting evidence from sources of varying credibility. This paper presents the first systematic evaluation of Retrieval-Augmented Generation (RAG) models for fact-checking in the presence of conflicting evidence. To support this study, we introduce \textbf{CONFACT} (\textbf{Con}flicting Evidence for \textbf{Fact}-Checking) (Dataset available at https://github.com/zoeyyes/CONFACT), a novel dataset comprising questions paired with conflicting information from various sources. Extensive experiments reveal critical vulnerabilities in state-of-the-art RAG methods, particularly in resolving conflicts stemming from differences in media source credibility. To address these challenges, we investigate strategies to integrate media background information into both the retrieval and generation stages. Our results show that effectively incorporating source credibility significantly enhances the ability of RAG models to resolve conflicting evidence and improve fact-checking performance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
