arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2307.13989cs.CLcs.LG
收藏

这不正确!语言生成系统的否定感知评估

This is not correct! Negation-aware Evaluation of Language Generation Systems

School for Computation, Information and Technology(计算、信息与技术学院);Technical University of Munich(慕尼黑工业大学)

Miriam Anschütz, Diego Miguel Lozano, Georg Groh

AI总结:

针对语言模型评估指标对否定不敏感的问题,提出否定感知评估指标NegBLEURT,构建CANNOT数据集进行微调,在否定句评估上大幅超越现有指标且保持基础性能。

摘要:

大型语言模型低估了否定对改变句子含义程度的影响。因此,基于这些模型的学习型评估指标对否定不敏感。本文提出了NegBLEURT,即BLEURT评估指标的否定感知版本。为此,我们设计了一个基于规则的句子否定工具,并用它创建了CANNOT否定评估数据集。基于该数据集,我们微调了句子Transformer和评估指标,以提高其否定敏感性。在现有基准上评估这些模型表明,我们微调后的模型在否定句上远胜现有指标,同时保持了其基础模型在其他扰动上的性能。

英文摘要:

Large language models underestimate the impact of negations on how much they change the meaning of a sentence. Therefore, learned evaluation metrics based on these models are insensitive to negations. In this paper, we propose NegBLEURT, a negation-aware version of the BLEURT evaluation metric. For that, we designed a rule-based sentence negation tool and used it to create the CANNOT negation evaluation dataset. Based on this dataset, we fine-tuned a sentence transformer and an evaluation metric to improve their negation sensitivity. Evaluating these models on existing benchmarks shows that our fine-tuned models outperform existing metrics on the negated sentences by far while preserving their base models' performances on other perturbations.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑