基于上下文学习的文本摘要多维评估
Multi-Dimensional Evaluation of Text Summarization with In-Context Learning
AI总结：
本文研究大型语言模型作为多维评估器的有效性，通过上下文学习免除对大型训练数据集的需求，实验表明该方法在文本摘要任务中具有竞争力，并在相关性和事实一致性等维度上达到最先进水平。
摘要：
自然语言生成（NLG）的评估是复杂且多维的。生成的文本可以从流畅性、连贯性、事实性或任何其他感兴趣的维度进行评估。大多数执行此类多维评估的框架都需要在大型手动或合成生成的数据集上进行训练。在本文中，我们研究了大型语言模型作为多维评估器的有效性，使用上下文学习，从而免除了对大型训练数据集的需求。我们的实验表明，对于文本摘要任务，基于上下文学习的评估器与已学习的评估框架具有竞争力，在相关性和事实一致性等维度上确立了最先进的水平。然后，我们分析了上下文示例的选择和数量等因素对性能的影响。最后，我们研究了基于上下文学习的评估器在评估由大型语言模型（如GPT-3）编写的零样本摘要时的有效性。
英文摘要：
Evaluation of natural language generation (NLG) is complex and multi-dimensional. Generated text can be evaluated for fluency, coherence, factuality, or any other dimensions of interest. Most frameworks that perform such multi-dimensional evaluation require training on large manually or synthetically generated datasets. In this paper, we study the efficacy of large language models as multi-dimensional evaluators using in-context learning, obviating the need for large training datasets. Our experiments show that in-context learning-based evaluators are competitive with learned evaluation frameworks for the task of text summarization, establishing state-of-the-art on dimensions such as relevance and factual consistency. We then analyze the effects of factors such as the selection and number of in-context examples on performance. Finally, we study the efficacy of in-context learning based evaluators in evaluating zero-shot summaries written by large language models such as GPT-3.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
