DecompEval：将生成文本评估为无监督分解式问答
DecompEval: Evaluating Generated Texts as Unsupervised Decomposed Question Answering
AI总结：
本文提出无监督指标DecompEval，通过指令式PLM问答并将文本质量问题分解为句子级子问题，以提升NLG评估的泛化性与可解释性。
摘要：
现有的自然语言生成（NLG）任务评估指标在泛化能力和可解释性方面面临挑战。具体而言，大多数表现良好的指标都需要在特定NLG任务和评估维度的评估数据集上进行训练，这可能导致对任务特定数据集的过拟合。此外，现有指标仅为每个维度提供一个评估分数，却不揭示用于解释该分数如何得出的证据。为应对这些挑战，我们提出了一种简单而有效的指标，名为DecompEval。该指标将NLG评估表述为指令式问答任务，并利用经过指令调优的预训练语言模型（PLM），且无需在评估数据集上训练，旨在增强泛化能力。为使评估过程更具可解释性，我们将所设计的关于生成文本质量的指令式问题分解为衡量每个句子质量的子问题。随后，将这些子问题及其由PLM生成的答案重新组合，作为获得评估结果的证据。实验结果表明，DecompEval在用于评估文本摘要和对话生成的未经训练指标中取得了最先进的性能，同时展现出较强的维度级/任务级泛化能力和可解释性。
英文摘要：
Existing evaluation metrics for natural language generation (NLG) tasks face the challenges on generalization ability and interpretability. Specifically, most of the well-performed metrics are required to train on evaluation datasets of specific NLG tasks and evaluation dimensions, which may cause over-fitting to task-specific datasets. Furthermore, existing metrics only provide an evaluation score for each dimension without revealing the evidence to interpret how this score is obtained. To deal with these challenges, we propose a simple yet effective metric called DecompEval. This metric formulates NLG evaluation as an instruction-style question answering task and utilizes instruction-tuned pre-trained language models (PLMs) without training on evaluation datasets, aiming to enhance the generalization ability. To make the evaluation process more interpretable, we decompose our devised instruction-style question about the quality of generated texts into the subquestions that measure the quality of each sentence. The subquestions with their answers generated by PLMs are then recomposed as evidence to obtain the evaluation result. Experimental results show that DecompEval achieves state-of-the-art performance in untrained metrics for evaluating text summarization and dialogue generation, which also exhibits strong dimension-level / task-level generalization ability and interpretability.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
