VERISCORE:评估长文本生成中可验证声明的事实性
VERISCORE: Evaluating the factuality of verifiable claims in long-form text generation
AI总结:
VERISCORE是一种评估长文本生成中可验证声明事实性的新指标,适用于包含可验证与不可验证内容的多样化任务,支持闭源与开源权重模型,揭示不同任务间事实性表现无必然相关性。
摘要:
现有的评估长文本事实性的指标,如FACTSCORE(Min等人,2023)和SAFE(Wei等人,2024),将输入文本分解为“原子声明”,并对照维基百科等知识库逐一验证。这些指标不适用于大多数生成任务,因为它们假设每个声明都是可验证的(即可以合理地被证明为真或假)。我们通过VERISCORE解决了这一问题,这是一种适用于包含可验证和不可验证内容的多样化长文本生成任务的指标。VERISCORE可以使用闭源或微调的开源权重语言模型有效实现,人工评估证实,在八个不同的长文本任务中,VERISCORE提取的声明比竞争方法更具合理性。我们使用VERISCORE评估了16个不同模型在多个长文本任务上的生成结果,发现虽然GPT-4o总体上是表现最佳的模型,但像Mixtral-8x22这样的开源权重模型正在缩小差距。我们表明,语言模型在一个任务(例如传记生成)上的VERISCORE得分不一定与其在另一个任务(例如长文本问答)上的得分相关,这突出了在事实密度不同的任务中扩展事实性评估的需求。
英文摘要:
Existing metrics for evaluating the factuality of long-form text, such as FACTSCORE (Min et al., 2023) and SAFE (Wei et al., 2024), decompose an input text into "atomic claims" and verify each against a knowledge base like Wikipedia. These metrics are not suitable for most generation tasks because they assume that every claim is verifiable (i.e., can plausibly be proven true or false). We address this issue with VERISCORE, a metric for diverse long-form generation tasks that contain both verifiable and unverifiable content. VERISCORE can be effectively implemented with either closed or fine-tuned open-weight language models, and human evaluation confirms that VERISCORE's extracted claims are more sensible than those from competing methods across eight different long-form tasks. We use VERISCORE to evaluate generations from 16 different models across multiple long-form tasks and find that while GPT-4o is the best-performing model overall, open-weight models such as Mixtral-8x22 are closing the gap. We show that an LM's VERISCORE on one task (e.g., biography generation) does not necessarily correlate to its VERISCORE on a different task (e.g., long-form QA), highlighting the need for expanding factuality evaluation across tasks with varying fact density.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。