Themis：一种具备灵活性与可解释性的无参考自然语言生成评价语言模型
Themis: A Reference-free NLG Evaluation Language Model with Flexibility and Interpretability
AI总结：
针对现有NLG评价方法依赖参考、灵活性不足的问题，本文构建了含人工和GPT-4标注的大规模语料库NLG-Eval，提出经多视角一致性验证与面向评分的偏好对齐训练的专用评价LLM Themis，实现无参考的灵活可解释评价，性能超越GPT-4等模型且泛化性好。
摘要：
自然语言生成（NLG）任务的评价是一个重要且历史悠久的研究领域。近期随着强大的大语言模型（LLM）的出现，部分研究转向基于LLM的自动评价方法，这类方法展现出成为继传统基于字符串和基于模型的指标之后新型评价范式的巨大潜力。然而，现有方法尽管性能有所提升，仍存在一些不足，例如依赖参考文本、评价灵活性有限。因此，本文精心构建了一个大规模NLG评价语料库NLG-Eval，该语料库同时包含人工标注和GPT-4标注，以缓解该领域相关数据匮乏的问题。此外，我们提出了专门用于NLG评价的LLM——Themis，该模型通过我们设计的多视角一致性验证和面向评分的偏好对齐方法训练而成。Themis能够在无参考文本的情况下开展灵活且可解释的评价，在多种NLG任务上表现出优异的评价性能，同时对未见过的任务具备良好的泛化能力，性能超越包括GPT-4在内的其他评价模型。
英文摘要：
The evaluation of natural language generation (NLG) tasks is a significant and longstanding research area. With the recent emergence of powerful large language models (LLMs), some studies have turned to LLM-based automatic evaluation methods, which demonstrate great potential to become a new evaluation paradigm following traditional string-based and model-based metrics. However, despite the improved performance of existing methods, they still possess some deficiencies, such as dependency on references and limited evaluation flexibility. Therefore, in this paper, we meticulously construct a large-scale NLG evaluation corpus NLG-Eval with annotations from both human and GPT-4 to alleviate the lack of relevant data in this field. Furthermore, we propose Themis, an LLM dedicated to NLG evaluation, which has been trained with our designed multi-perspective consistency verification and rating-oriented preference alignment methods. Themis can conduct flexible and interpretable evaluations without references, and it exhibits superior evaluation performance on various NLG tasks, simultaneously generalizing well to unseen tasks and surpassing other evaluation models, including GPT-4.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
