CoAScore：面向自然语言生成评估的方面链提示方法
CoAScore: Chain-of-Aspects Prompting for NLG Evaluation
AI总结：
针对现有大语言模型NLG评估忽略多方面关联性的问题，提出CoAScore评估指标，通过方面链提示框架利用多方面知识优化目标方面评估，在多任务多方面上显著提升与人类判断的相关性，优于现有无监督指标。
摘要：
近年来，自然语言生成（NLG）评估已从单一方面范式转向多方面范式，从而能够实现更精准的评估。大语言模型（LLM）在各类NLG评估任务上均取得了优异性能。然而，现有研究通常采用LLM对不同方面进行独立评估，这在很大程度上忽略了各方面之间丰富的关联性。为填补这一研究空白，本研究提出了一种名为CoAScore的NLG评估指标。在LLM的支撑下，CoAScore在评估某一特定方面的质量时，通过CoA（方面链，Chain-of-Aspects）提示框架利用多方面知识。具体而言，对于待评估的给定目标方面，我们首先提示LLM生成一条与目标方面相关、且可能对评估有帮助的方面链；随后，我们收集每个生成方面的评估分数；最后，利用这些方面的知识来优化目标方面的评估效果。我们在五项NLG评估任务（如摘要生成、对话回复生成等）和九个评估方面（如整体质量、相关性、连贯性等）上对CoAScore进行了评估。实验结果表明，与单一方面独立评估相比，CoAScore与人类判断的相关性更高。无论是评估整体质量还是其他方面，这一提升效果都显著优于现有的无监督评估指标。我们还开展了大量消融实验，验证了CoAScore框架中三个阶段的有效性，并通过案例研究展示了LLM在这些阶段中的运行表现。我们的代码和脚本已公开可用。
英文摘要：
Recently, natural language generation (NLG) evaluation has shifted from a single-aspect to a multi-aspect paradigm, allowing for a more accurate assessment. Large language models (LLMs) achieve superior performance on various NLG evaluation tasks. However, current work often employs the LLM to independently evaluate different aspects, which largely ignores the rich correlation between various aspects. To fill this research gap, in this work, we propose an NLG evaluation metric called CoAScore. Powered by LLMs, the CoAScore utilizes multi-aspect knowledge through a CoA (\textbf{C}hain-\textbf{o}f-\textbf{A}spects) prompting framework when assessing the quality of a certain aspect. Specifically, for a given aspect to evaluate, we first prompt the LLM to generate a chain of aspects that are relevant to the target aspect and could be useful for the evaluation. We then collect evaluation scores for each generated aspect, and finally, leverage the knowledge of these aspects to improve the evaluation of the target aspect. We evaluate CoAScore across five NLG evaluation tasks (e.g., summarization, dialog response generation, etc) and nine aspects (e.g., overall quality, relevance, coherence, etc). Our experimental findings highlight that, in comparison to individual aspect evaluation, CoAScore exhibits a higher correlation with human judgments. This improvement significantly outperforms existing unsupervised evaluation metrics, whether for assessing overall quality or other aspects. We also conducted extensive ablation studies to validate the effectiveness of the three stages within the CoAScore framework and conducted case studies to show how the LLM performs in these stages. Our code and scripts are available.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
