GUMSum：面向英语抽象式摘要的多体裁数据与评估
GUMSum: Multi-Genre Data and Evaluation for English Abstractive Summarization
AI总结：
GUMSum构建了12种体裁的英语摘要数据集，评估发现GPT3虽流畅但仍有幻觉且不如人类，并揭示了不同方法生成摘要的错误类型与挑战。
摘要：
使用预训练语言模型进行自动摘要已产生令人印象深刻的流畅结果，但容易出现“幻觉”、在非新闻体裁上表现不佳，以及输出并非严格意义上的摘要。针对ACL 2023年的“现实检验”主题，我们提出了GUMSum，这是一个规模虽小但精心构建的英语摘要数据集，涵盖12种书面和口语体裁，用于评估抽象式摘要。摘要受到高度约束，重点关注替代潜力、事实性和忠实性。我们提出了指南，并评估了人类一致性以及对近期系统输出的主观判断，将通用领域未调优方法、微调方法和基于提示的方法与人类表现进行比较。结果表明，尽管GPT3取得了令人印象深刻的分数，但其表现仍不如人类，且各体裁质量参差不齐。人类判断揭示了监督式、提示式及人工生成的摘要中不同类型的错误，揭示了生成一篇好摘要所面临的挑战。
英文摘要：
Automatic summarization with pre-trained language models has led to impressively fluent results, but is prone to 'hallucinations', low performance on non-news genres, and outputs which are not exactly summaries. Targeting ACL 2023's 'Reality Check' theme, we present GUMSum, a small but carefully crafted dataset of English summaries in 12 written and spoken genres for evaluation of abstractive summarization. Summaries are highly constrained, focusing on substitutive potential, factuality, and faithfulness. We present guidelines and evaluate human agreement as well as subjective judgments on recent system outputs, comparing general-domain untuned approaches, a fine-tuned one, and a prompt-based approach, to human performance. Results show that while GPT3 achieves impressive scores, it still underperforms humans, with varying quality across genres. Human judgments reveal different types of errors in supervised, prompted, and human-generated summaries, shedding light on the challenges of producing a good summary.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
