BMX:用可解释性提升自然语言生成指标
BMX: Boosting Natural Language Generation Metrics with Explainability
AI总结:
提出 BMX 方法,通过将特征重要性解释转换为片段级分数并与原始指标结合,有效提升了自然语言生成评估指标的性能,在摘要任务上提升显著。
摘要:
最先进的自然语言生成评估指标基于黑盒语言模型。因此,近期研究考虑其可解释性,旨在提升人类可理解性及指标分析(包括失败案例)。相比之下,我们提出的方法 BMX: Boosting Natural Language Generation Metrics with explainability 明确利用解释来提升指标性能。具体而言,我们将特征重要性解释视为词级分数,并通过幂平均将其转换为片段级分数。随后,将该片段级分数与原始指标结合以获得更优指标。测试表明,在机器翻译和摘要数据集上多项指标均有所提升。机器翻译的提升较小,但在摘要任务上提升显著。值得注意的是,使用 LIME 解释器和预选参数的 BMX 在 SummEval 的系统级评估中,Spearman 相关系数平均提升了 0.087 个点。
英文摘要:
State-of-the-art natural language generation evaluation metrics are based on black-box language models. Hence, recent works consider their explainability with the goals of better understandability for humans and better metric analysis, including failure cases. In contrast, our proposed method BMX: Boosting Natural Language Generation Metrics with explainability explicitly leverages explanations to boost the metrics' performance. In particular, we perceive feature importance explanations as word-level scores, which we convert, via power means, into a segment-level score. We then combine this segment-level score with the original metric to obtain a better metric. Our tests show improvements for multiple metrics across MT and summarization datasets. While improvements in machine translation are small, they are strong for summarization. Notably, BMX with the LIME explainer and preselected parameters achieves an average improvement of 0.087 points in Spearman correlation on the system-level evaluation of SummEval.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。