GPT-3时代的新闻摘要与评估
News Summarization and Evaluation in the Era of GPT-3
AI总结：
本文研究GPT-3对新闻摘要的影响，发现人类更偏好GPT-3摘要且其事实性更好，同时指出现有自动指标无法可靠评估GPT-3摘要，并发布了相关语料库和人类偏好判断数据。
摘要：
最近，像GPT-3这样的大型语言模型的提示成功导致了NLP研究的范式转变。在本文中，我们研究其对文本摘要的影响，重点关注新闻摘要这一经典基准领域。首先，我们调查GPT-3与在大型摘要数据集上训练的微调模型相比表现如何。我们表明，不仅人类压倒性地偏好仅使用任务描述提示的GPT-3摘要，而且这些摘要也不会受到特定于数据集的常见问题（如事实性差）的困扰。接下来，我们研究这对评估意味着什么，特别是黄金标准测试集的作用。我们的实验表明，基于参考和无参考的自动指标都无法可靠地评估GPT-3摘要。最后，我们在通用摘要之外的设置上评估模型，特别是基于关键字的摘要，并展示主流微调方法与提示的比较。为了支持进一步研究，我们发布：(a)一个包含来自4个标准摘要基准的微调和基于提示的模型的10K生成摘要的语料库，(b)1K比较通用和基于关键字摘要的不同系统的人类偏好判断。
英文摘要：
The recent success of prompting large language models like GPT-3 has led to a paradigm shift in NLP research. In this paper, we study its impact on text summarization, focusing on the classic benchmark domain of news summarization. First, we investigate how GPT-3 compares against fine-tuned models trained on large summarization datasets. We show that not only do humans overwhelmingly prefer GPT-3 summaries, prompted using only a task description, but these also do not suffer from common dataset-specific issues such as poor factuality. Next, we study what this means for evaluation, particularly the role of gold standard test sets. Our experiments show that both reference-based and reference-free automatic metrics cannot reliably evaluate GPT-3 summaries. Finally, we evaluate models on a setting beyond generic summarization, specifically keyword-based summarization, and show how dominant fine-tuning approaches compare to prompting. To support further research, we release: (a) a corpus of 10K generated summaries from fine-tuned and prompt-based models across 4 standard summarization benchmarks, (b) 1K human preference judgments comparing different systems for generic- and keyword-based summarization.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
