arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2402.18678cs.CL
收藏

RORA:稳健的自由文本理由评估

RORA: Robust Free-Text Rationale Evaluation

Johns Hopkins University(约翰斯·霍普金斯大学)

Zhengping Jiang, Yining Lu, Hanjie Chen, Daniel Khashabi, Benjamin Van Durme, Anqi Liu

AI总结:

本文针对自由文本理由易发生标签泄露而现有指标评估失效的问题,提出基于稳健条件 V-information 的 RORA,以量化理由新增信息,并证明其更稳健且更贴合人类判断。

摘要:

自由文本理由在可解释 NLP 中发挥着关键作用,可弥合模型决策背后的知识与推理差距。然而,由于潜在推理路径多样且相应缺乏确定的真值,对其进行评估仍是一项挑战。现有评估指标依赖于理由对目标标签的支持程度,但我们发现,这些指标在评估无意中泄露标签的理由时存在不足。为解决该问题,我们提出 RORA,一种针对标签泄露的稳健自由文本理由评估方法。RORA 量化理由为证明标签合理性而提供的新信息。这是通过利用一个对可被小模型利用的泄露特征具有稳健性的预测族来评估条件 V-information 实现的。在评估人工编写、合成或模型生成的理由时,RORA 始终优于现有方法,尤其展现出对标签泄露的稳健性。我们还表明,RORA 与人类判断高度一致,可在各类自由文本理由中提供更可靠、更准确的度量。

英文摘要:

Free-text rationales play a pivotal role in explainable NLP, bridging the knowledge and reasoning gaps behind a model's decision-making. However, due to the diversity of potential reasoning paths and a corresponding lack of definitive ground truth, their evaluation remains a challenge. Existing evaluation metrics rely on the degree to which a rationale supports a target label, but we find these fall short in evaluating rationales that inadvertently leak the labels. To address this problem, we propose RORA, a Robust free-text Rationale evaluation against label leakage. RORA quantifies the new information supplied by a rationale to justify the label. This is achieved by assessing the conditional V-information \citep{hewitt-etal-2021-conditional} with a predictive family robust against leaky features that can be exploited by a small model. RORA consistently outperforms existing approaches in evaluating human-written, synthetic, or model-generated rationales, particularly demonstrating robustness against label leakage. We also show that RORA aligns well with human judgment, providing a more reliable and accurate measurement across diverse free-text rationales.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑