arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2207.00779cs.CLcs.AIcs.LG
收藏

FRAME:评估自由文本文档的文档标签一致性

FRAME: Evaluating Rationale-Label Consistency Metrics for Free-Text Rationales

University of Southern California(南加州大学);Meta AI

Aaron Chan, Shaoliang Nie, Liang Tan, Xiaochang Peng, Hamed Firooz, Maziar Sanjabi, Xiang Ren

摘要:

遵循人类如何交流,自由文本理论旨在使用自然语言来解释神经语言模型(LM)行为。然而,自由文本基本原理的不受约束的性质使它们容易产生幻觉,因此重要的是要有自由文本基本原理质量的指标。现有的自由文本基本原理度量标准测量基本原理与LM的预测标签的一致性,但没有用于评估此类度量标准可靠性的协议。因此,我们提出框架,一个框架,用于评估自由文本的理由标签一致性(RLC)指标。FRAME基于三个公理:(1)好的度量应该产生最高分数的参考理由,最大化RLC的建设;(2)好的度量应该是适当的敏感性的语义扰动的理由;(3)好的度量应该是强大的LM的任务性能的变化。在三个文本分类数据集,我们表明,现有的RLC指标不能满足所有三个框架公理,因为它们是通过模型预训练,混淆度量的信号。然后,我们引入了一个非预训练RLC度量,它在(1)和(3)上的性能大大优于基线,而在(2)上的性能具有竞争力。最后,我们讨论了使用RLC评估自由文本的理由的局限性。

英文摘要:

Following how humans communicate, free-text rationales aim to use natural language to explain neural language model (LM) behavior. However, free-text rationales' unconstrained nature makes them prone to hallucination, so it is important to have metrics for free-text rationale quality. Existing free-text rationale metrics measure how consistent the rationale is with the LM's predicted label, but there is no protocol for assessing such metrics' reliability. Thus, we propose FRAME, a framework for evaluating rationale-label consistency (RLC) metrics for free-text rationales. FRAME is based on three axioms: (1) good metrics should yield highest scores for reference rationales, which maximize RLC by construction; (2) good metrics should be appropriately sensitive to semantic perturbation of rationales; and (3) good metrics should be robust to variation in the LM's task performance. Across three text classification datasets, we show that existing RLC metrics cannot satisfy all three FRAME axioms, since they are implemented via model pretraining which muddles the metric's signal. Then, we introduce a non-pretraining RLC metric that greatly outperforms baselines on (1) and (3), while performing competitively on (2). Finally, we discuss the limitations of using RLC to evaluate free-text rationales.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑