X-Eval：通过带辅助评估维度的增强指令微调实现可泛化的多维度文本评估
X-Eval: Generalizable Multi-aspect Text Evaluation via Augmented Instruction Tuning with Auxiliary Evaluation Aspects
AI总结：
针对多维度文本评估需泛化到未见维度的难题，提出两阶段指令微调框架X-Eval，搭配定制数据集AspectInstruct，可让轻量模型达到与GPT-4等先进评估器相当甚至更高的人类判断相关性。
摘要：
自然语言生成（NLG）通常需要从多个维度（如一致性、自然度）评估生成文本，以获得全面的评价结果。然而，多维度评估仍具挑战性，因为它要求评估器能泛化到任意给定的评估维度，即便该维度在训练过程中未出现过。本文提出X-Eval，这是一个两阶段指令微调框架，可对终端用户自定义的已见和未见维度的文本进行评估。X-Eval包含两个学习阶段：普通指令微调阶段，用于提升模型遵循评估指令的能力；增强指令微调阶段，利用细粒度评估维度之间的关联来更好地评估文本质量。为支撑X-Eval的训练，我们构建了AspectInstruct，这是首个专为多维度NLG评估定制的指令微调数据集，涵盖27个不同的评估维度，包含65项任务。为提升任务多样性，我们设计了一种增强策略，将人工评分标注转换为多种形式的NLG评估任务，包括打分、比较、排序以及布尔问答。我们在对话生成、摘要生成、数据到文本生成三类核心NLG任务上开展了大量实验，并结合21个维度进行元评估，结果表明，即便使用轻量级语言模型，X-Eval也能使其与GPT-4等最先进的NLG评估器相比，达到与人类判断相当（甚至更高）的相关性。
英文摘要：
Natural Language Generation (NLG) typically involves evaluating the generated text in various aspects (e.g., consistency and naturalness) to obtain a comprehensive assessment. However, multi-aspect evaluation remains challenging as it may require the evaluator to generalize to any given evaluation aspect even if it's absent during training. In this paper, we introduce X-Eval, a two-stage instruction tuning framework to evaluate the text in both seen and unseen aspects customized by end users. X-Eval consists of two learning stages: the vanilla instruction tuning stage that improves the model's ability to follow evaluation instructions, and an enhanced instruction tuning stage that exploits the connections between fine-grained evaluation aspects to better assess text quality. To support the training of X-Eval, we collect AspectInstruct, the first instruction tuning dataset tailored for multi-aspect NLG evaluation spanning 27 diverse evaluation aspects with 65 tasks. To enhance task diversity, we devise an augmentation strategy that converts human rating annotations into diverse forms of NLG evaluation tasks, including scoring, comparison, ranking, and Boolean question answering. Extensive experiments across three essential categories of NLG tasks: dialogue generation, summarization, and data-to-text coupled with 21 aspects in meta-evaluation, demonstrate that our X-Eval enables even a lightweight language model to achieve a comparable if not higher correlation with human judgments compared to the state-of-the-art NLG evaluators, such as GPT-4.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
