一个提示统领一切：用于观点摘要评估的大语言模型
One Prompt To Rule Them All: LLMs for Opinion Summary Evaluation
AI总结：
本文首次在观点摘要评估领域研究大语言模型作为无参考评估器，发布SUMMEVAL-OP数据集，并提出Op-I-Prompt提示方法，实现与人类判断0.70的平均Spearman相关性，优于所有先前方法。
摘要：
使用传统的基于参考的指标评估观点摘要很少能提供整体性评价，并且已被证明与人类判断的相关性相对较低。近期研究建议使用大语言模型（LLMs）作为自然语言生成（NLG）评估的无参考指标，然而它们在观点摘要评估方面仍未得到探索。此外，有限的观点摘要评估数据集阻碍了进展。为了解决这一问题，我们发布了SUMMEVAL-OP数据集，该数据集涵盖与观点摘要评估相关的7个维度：流畅性、连贯性、相关性、忠实性、方面覆盖度、情感一致性和具体性。我们研究了Op-I-Prompt（一种与维度无关的提示）和Op-Prompts（一组与维度相关的提示）用于观点摘要评估。实验表明，Op-I-Prompt成为评估观点摘要的良好替代方案，与人类评估实现了平均0.70的Spearman相关性，优于所有先前的方法。据我们所知，我们是首个在观点摘要领域同时研究闭源和开源模型上大语言模型作为评估者的工作。
英文摘要：
Evaluation of opinion summaries using conventional reference-based metrics rarely provides a holistic evaluation and has been shown to have a relatively low correlation with human judgments. Recent studies suggest using Large Language Models (LLMs) as reference-free metrics for NLG evaluation, however, they remain unexplored for opinion summary evaluation. Moreover, limited opinion summary evaluation datasets inhibit progress. To address this, we release the SUMMEVAL-OP dataset covering 7 dimensions related to the evaluation of opinion summaries: fluency, coherence, relevance, faithfulness, aspect coverage, sentiment consistency, and specificity. We investigate Op-I-Prompt a dimension-independent prompt, and Op-Prompts, a dimension-dependent set of prompts for opinion summary evaluation. Experiments indicate that Op-I-Prompt emerges as a good alternative for evaluating opinion summaries achieving an average Spearman correlation of 0.70 with humans, outperforming all previous approaches. To the best of our knowledge, we are the first to investigate LLMs as evaluators on both closed-source and open-source models in the opinion summarization domain.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
