arXiv 2502.089092025-02-13cs.CLcs.AI

迈向真实世界主张的自动化事实核查：利用大语言模型探索任务制定与评估

Towards Automated Fact-Checking of Real-World Claims: Exploring Task Formulation and Assessment with LLMs

Technische Universität Berlin（柏林工业大学）；Deutsches Forschungszentrum für Künstliche Intelligenz（德国人工智能研究中心）；National Institute of Informatics（信息学研究所）；Harz University of Applied Sciences（哈尔茨应用技术大学）

Premtim Sahitaj, Iffat Maab, Junichi Yamagishi, Jawan Kolanowski, Sebastian Möller, Vera Schmitt

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对传统人工事实核查慢且资源密集的问题，本研究用不同规模Llama-3模型在多标签方案下建立自动化事实核查基准，结合检索证据评估，发现大模型未微调性能优于小模型，小模型单样本性能可比微调小模型，为真实世界主张自动化核查提供参考。

摘要：

事实核查对于应对日益增多的错误信息至关重要。传统事实核查依赖人工分析验证主张，但速度慢且资源密集。本研究建立了使用大语言模型（LLMs）的自动化事实核查（AFC）基准对比，覆盖多种标签方案（二分类、三分类、五分类），并通过在结构化设置中整合分析、 verdict 分类和解释来扩展传统主张验证，为真实世界主张提供全面依据。我们在通过受限网络搜索检索的证据上，评估了不同规模（3B、8B、70B）的 Llama-3 模型，使用 TIGERScore 作为无参考评估指标对依据进行评分。结果显示，更大的 LLMs 在未微调的情况下，分类准确率和依据质量持续优于更小的 LLMs；小 LLMs 在单样本场景下的任务性能与大上下文规模的微调小语言模型（SLMs）相当，而大 LLMs 持续超越它们；证据整合提升了所有模型的性能，大 LLMs 受益最多。区分细微标签仍具挑战，强调需进一步探索标签方案及与证据的对齐。我们的发现证明了检索增强型 LLM 自动化事实核查的潜力。

英文摘要：

Fact-checking is necessary to address the increasing volume of misinformation. Traditional fact-checking relies on manual analysis to verify claims, but it is slow and resource-intensive. This study establishes baseline comparisons for Automated Fact-Checking (AFC) using Large Language Models (LLMs) across multiple labeling schemes (binary, three-class, five-class) and extends traditional claim verification by incorporating analysis, verdict classification, and explanation in a structured setup to provide comprehensive justifications for real-world claims. We evaluate Llama-3 models of varying sizes (3B, 8B, 70B) on 17,856 claims collected from PolitiFact (2007-2024) using evidence retrieved via restricted web searches. We utilize TIGERScore as a reference-free evaluation metric to score the justifications. Our results show that larger LLMs consistently outperform smaller LLMs in classification accuracy and justification quality without fine-tuning. We find that smaller LLMs in a one-shot scenario provide comparable task performance to fine-tuned Small Language Models (SLMs) with large context sizes, while larger LLMs consistently surpass them. Evidence integration improves performance across all models, with larger LLMs benefiting most. Distinguishing between nuanced labels remains challenging, emphasizing the need for further exploration of labeling schemes and alignment with evidences. Our findings demonstrate the potential of retrieval-augmented AFC with LLMs.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。