揭示 NLG 评估器的阿喀琉斯之踵：一个由大型语言模型驱动的统一对抗框架
Unveiling the Achilles' Heel of NLG Evaluators: A Unified Adversarial Framework Driven by Large Language Models
AI总结：
提出由 LLM 驱动的黑盒对抗框架 AdvEval，通过黄金与受害评估器反馈自动构造分歧样本，揭示多种 NLG 评估指标在多任务上的鲁棒性缺陷。
摘要：
自然语言生成（NLG）系统的自动评估是一项长期存在的挑战。近期研究强调了多种与人类评估具有良好一致性的神经指标。然而，由于为不同 NLG 评估任务获取对抗数据存在独特挑战，这些评估器面对对抗扰动时的鲁棒性在很大程度上仍未得到充分探索。为解决该问题，我们提出 AdvEval，这是一种针对 NLG 评估器的新型黑盒对抗框架。AdvEval 经过专门设计，用于生成会导致人类与受害评估器之间产生强烈分歧的数据。具体而言，受大型语言模型（LLM）近期在文本生成和评估方面成功的启发，我们采用强大的 LLM 同时充当数据生成器和黄金评估器。对抗数据会利用来自黄金评估器和受害评估器的反馈进行自动优化。我们在 12 个受害评估器和 11 个 NLG 数据集上开展实验，任务涵盖对话、摘要和问题评估。结果表明，AdvEval 能够导致多种受害指标的性能显著下降，从而验证了其有效性。
英文摘要：
The automatic evaluation of natural language generation (NLG) systems presents a long-lasting challenge. Recent studies have highlighted various neural metrics that align well with human evaluations. Yet, the robustness of these evaluators against adversarial perturbations remains largely under-explored due to the unique challenges in obtaining adversarial data for different NLG evaluation tasks. To address the problem, we introduce AdvEval, a novel black-box adversarial framework against NLG evaluators. AdvEval is specially tailored to generate data that yield strong disagreements between human and victim evaluators. Specifically, inspired by the recent success of large language models (LLMs) in text generation and evaluation, we adopt strong LLMs as both the data generator and gold evaluator. Adversarial data are automatically optimized with feedback from the gold and victim evaluator. We conduct experiments on 12 victim evaluators and 11 NLG datasets, spanning tasks including dialogue, summarization, and question evaluation. The results show that AdvEval can lead to significant performance degradation of various victim metrics, thereby validating its efficacy.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
