arXiv 2311.090002024-04-16cs.CL

Factcheck-Bench：面向自动事实核查器的细粒度评估基准

Factcheck-Bench: Fine-Grained Evaluation Benchmark for Automatic Fact-checkers

MBZUAI（穆罕默德·本·扎耶德人工智能大学）；University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；University of Copenhagen（哥本哈根大学）；University of California, Santa Barbara（加州大学圣巴巴拉分校）；Skolkovo Institute of Science and Technology（斯科尔科沃科学技术研究院）

Yuxia Wang, Revanth Gangi Reddy, Zain Muhammad Mujahid, Arnav Arora, Aleksandr Rubashevskii, Jiahui Geng, Osama Mohammed Afzal, Liangming Pan, Nadav Borenstein,… 展开作者

Yuxia Wang, Revanth Gangi Reddy, Zain Muhammad Mujahid, Arnav Arora, Aleksandr Rubashevskii, Jiahui Geng, Osama Mohammed Afzal, Liangming Pan, Nadav Borenstein, Aditya Pillai, Isabelle Augenstein, Iryna Gurevych, Preslav Nakov

备注：: 30 pages, 13 figures

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出多阶段事实性标注方案，并构建覆盖声明、句子和文档三层粒度的 Factcheck-Bench，用于评估自动事实核查系统。

摘要：

大型语言模型（LLM）在各类现实应用中的使用日益增加，因此需要有机制来验证其输出的事实准确性。在本研究中，我们提出了一种用于标注 LLM 生成回复事实性的整体端到端解决方案，其中包含一套多阶段标注方案，旨在生成有关 LLM 输出中可验证性和事实不一致性的详细标签。我们还进一步构建了一个开放领域、文档级的事实性基准，粒度分为三个层级：声明、句子和文档，旨在促进自动事实核查系统的评估。初步实验表明，FacTool、FactScore 和 Perplexity.ai 在识别虚假声明方面存在困难，而基于 GPT-4 的该标注解决方案取得了最佳 F1=0.63。标注工具、基准和代码可在 https://github.com/yuxiaw/Factcheck-GPT 获取。

英文摘要：

The increased use of large language models (LLMs) across a variety of real-world applications calls for mechanisms to verify the factual accuracy of their outputs. In this work, we present a holistic end-to-end solution for annotating the factuality of LLM-generated responses, which encompasses a multi-stage annotation scheme designed to yield detailed labels concerning the verifiability and factual inconsistencies found in LLM outputs. We further construct an open-domain document-level factuality benchmark in three-level granularity: claim, sentence and document, aiming to facilitate the evaluation of automatic fact-checking systems. Preliminary experiments show that FacTool, FactScore and Perplexity.ai are struggling to identify false claims, with the best F1=0.63 by this annotation solution based on GPT-4. Annotation tool, benchmark and code are available at https://github.com/yuxiaw/Factcheck-GPT.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。