OpenFactCheck：构建、基准测试定制化事实核查系统并评估声明与大语言模型的事实性
OpenFactCheck: Building, Benchmarking Customized Fact-Checking Systems and Evaluating the Factuality of Claims and LLMs
AI总结：
针对LLM输出事实性评估难、不同研究基准不统一的问题，研究提出OpenFactCheck统一框架，含定制核查、LLM评估、核查器评估三大模块，可构建定制事实核查系统并开展多维度事实性评估。
摘要：
大语言模型（LLM）在各类现实应用中的使用日益广泛，亟需验证其输出事实准确性的机制。当前难点在于评估开放领域中自由形式回复的事实性，且不同研究采用的评估基准与度量标准各不相同，导致结果难以比较，阻碍了领域的后续发展。为缓解这些问题，我们提出OpenFactCheck——一个统一框架，用于构建定制化自动事实核查系统、对其准确率进行基准测试、评估LLM的事实性以及验证文档中的声明。OpenFactCheck包含三个模块：（i）CUSTCHECKER（定制核查器）支持用户轻松定制自动事实核查器，验证文档与声明的事实正确性；（ii）LLMEVAL（大语言模型评估）是统一评估框架，可从多个维度公平评估LLM的事实性能力；（iii）CHECKEREVAL（核查器评估）是可扩展方案，借助人工标注数据集衡量自动事实核查器验证结果的可靠性。数据与代码已公开于https://github.com/yuxiaw/openfactcheck。
英文摘要：
The increased use of large language models (LLMs) across a variety of real-world applications calls for mechanisms to verify the factual accuracy of their outputs. Difficulties lie in assessing the factuality of free-form responses in open domains. Also, different papers use disparate evaluation benchmarks and measurements, which renders them hard to compare and hampers future progress. To mitigate these issues, we propose OpenFactCheck, a unified framework for building customized automatic fact-checking systems, benchmarking their accuracy, evaluating factuality of LLMs, and verifying claims in a document. OpenFactCheck consists of three modules: (i) CUSTCHECKER allows users to easily customize an automatic fact-checker and verify the factual correctness of documents and claims, (ii) LLMEVAL, a unified evaluation framework assesses LLM's factuality ability from various perspectives fairly, and (iii) CHECKEREVAL is an extensible solution for gauging the reliability of automatic fact-checkers' verification results using human-annotated datasets. Data and code are publicly available at https://github.com/yuxiaw/openfactcheck.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
