arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2408.11832cs.CLcs.AI
收藏

OpenFactCheck:一个面向大语言模型事实性评估的统一框架

OpenFactCheck: A Unified Framework for Factuality Evaluation of LLMs

MBZUAI;Monash University(墨尔本大学);Sofia University(索菲亚大学)

Hasan Iqbal, Yuxia Wang, Minghan Wang, Georgi Georgiev, Jiahui Geng, Iryna Gurevych, Preslav Nakov

AI总结:

针对大语言模型事实性评估缺乏统一标准、研究难以比较的问题,研究人员提出包含响应评估、模型整体事实性评估、事实验证系统评估三大模块的统一框架OpenFactCheck,已开源并提供多形态公开版本。

摘要:

大语言模型(LLM)在各类真实应用中的使用日益广泛,但LLM常产生幻觉,因此需要自动化工具来检验其输出的事实准确性。这项工作的难点在于需要评估自由形式开放域回答的事实性。尽管该领域已有大量研究,但不同论文采用的评估基准和度量标准各不相同,导致研究间难以相互比较,阻碍了领域的后续发展。为解决这些问题,我们开发了统一框架OpenFactCheck,它包含三个模块:(i)RESPONSEEVAL(响应评估模块),支持用户轻松定制自动化事实验证系统,并使用该系统评估输入文档中所有主张的事实性;(ii)LLMEVAL(大语言模型评估模块),用于评估LLM的整体事实性;(iii)CHECKEREVAL(验证器评估模块),用于评估自动化事实验证系统的性能。OpenFactCheck已开源(项目地址:https://github.com/mbzuai-nlp/openfactcheck),并以Python库(地址:https://pypi.org/project/openfactcheck/)和网页服务(地址:http://app.openfactcheck.com)的形式公开发布,系统介绍视频可在https://youtu.be/-i9VKL0HleI 查看。

英文摘要:

The increased use of large language models (LLMs) across a variety of real-world applications calls for automatic tools to check the factual accuracy of their outputs, as LLMs often hallucinate. This is difficult as it requires assessing the factuality of free-form open-domain responses. While there has been a lot of research on this topic, different papers use different evaluation benchmarks and measures, which makes them hard to compare and hampers future progress. To mitigate these issues, we developed OpenFactCheck, a unified framework, with three modules: (i) RESPONSEEVAL, which allows users to easily customize an automatic fact-checking system and to assess the factuality of all claims in an input document using that system, (ii) LLMEVAL, which assesses the overall factuality of an LLM, and (iii) CHECKEREVAL, a module to evaluate automatic fact-checking systems. OpenFactCheck is open-sourced (https://github.com/mbzuai-nlp/openfactcheck) and publicly released as a Python library (https://pypi.org/project/openfactcheck/) and also as a web service (http://app.openfactcheck.com). A video describing the system is available at https://youtu.be/-i9VKL0HleI.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑