arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2212.07919cs.CLcs.LG
收藏

ROSCOE:一套用于评分逐步推理的指标

ROSCOE: A Suite of Metrics for Scoring Step-by-Step Reasoning

Meta AI Research(Meta人工智能研究院)

Olga Golovneva, Moya Chen, Spencer Poff, Martin Corredor, Luke Zettlemoyer, Maryam Fazel-Zarandi, Asli Celikyilmaz

AI总结:

本文提出ROSCOE,一套可解释的无监督自动评分指标,用于评估大型语言模型逐步推理步骤的正确性,在语义一致性、逻辑性等多维度上持续优于基线指标。

摘要:

当提示大型语言模型生成逐步推理来证明其最终答案时,它们在下游任务上的性能会有所提升。这些推理步骤极大地改善了模型的可解释性和验证,但如果没有可靠的自动评估方法,客观地研究其正确性(独立于最终答案)是很困难的。我们简直不知道所陈述的推理步骤实际上有多少能支持最终的任务预测。在这项工作中,我们提出了ROSCOE,一套可解释的、无监督的自动评分指标,它改进并扩展了以往的文本生成评估指标。为了评估ROSCOE与基线指标的表现,我们设计了一种推理错误类型学,并在常用的推理数据集上收集了合成和人工评估分数。与现有指标不同,ROSCOE通过利用逐步推理依据的属性,能够衡量语义一致性、逻辑性、信息量、流畅性和事实性等特征。我们在5个人工标注和6个程序化扰动的诊断数据集上实证验证了我们指标的强度,这些数据集涵盖了需要推理技能的多种任务,并表明ROSCOE能够持续优于基线指标。

英文摘要:

Large language models show improved downstream task performance when prompted to generate step-by-step reasoning to justify their final answers. These reasoning steps greatly improve model interpretability and verification, but objectively studying their correctness (independent of the final answer) is difficult without reliable methods for automatic evaluation. We simply do not know how often the stated reasoning steps actually support the final end task predictions. In this work, we present ROSCOE, a suite of interpretable, unsupervised automatic scores that improve and extend previous text generation evaluation metrics. To evaluate ROSCOE against baseline metrics, we design a typology of reasoning errors and collect synthetic and human evaluation scores on commonly used reasoning datasets. In contrast with existing metrics, ROSCOE can measure semantic consistency, logicality, informativeness, fluency, and factuality - among other traits - by leveraging properties of step-by-step rationales. We empirically verify the strength of our metrics on five human annotated and six programmatically perturbed diagnostics datasets - covering a diverse set of tasks that require reasoning skills and show that ROSCOE can consistently outperform baseline metrics.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑