跨领域零样本摘要器的事实性评估
Evaluating the Factuality of Zero-shot Summarizers Across Varied Domains
AI总结：
本研究评估零样本摘要器在生物医学和法律等专业领域的事实性，通过专家标注错误并分析预训练语料领域普遍性的影响，发布标注数据集以促进事实准确摘要研究。
摘要：
近期研究表明，大型语言模型（LLMs）能够零样本（即无需显式监督）生成摘要，在人类评估下，这些摘要往往与人工撰写的参考摘要相当甚至更受青睐。然而，先前的工作几乎 exclusively 聚焦于新闻文章摘要的评估。零样本摘要器在其他（可能更专业的）领域表现如何？在本工作中，我们评估了跨专业领域（包括生物医学文章和法律法案，以及作为参考的标准新闻基准）的零样本生成摘要。我们特别关注输出的事实性。我们获取领域专家的标注，以识别摘要中的不一致之处，并系统性地对这些错误进行分类。我们分析预训练语料中某一领域的普遍性是否影响该领域文章生成摘要的抽取性和忠实性。我们发布所有收集的标注，以促进超越新闻文章的事实准确摘要测量与实现方面的进一步研究。数据集可从 https://github.com/sanjanaramprasad/zero_shot_faceval_domains 下载。
英文摘要：
Recent work has shown that large language models (LLMs) are capable of generating summaries zero-shot (i.e., without explicit supervision) that, under human assessment, are often comparable or even preferred to manually composed reference summaries. However, this prior work has focussed almost exclusively on evaluating news article summarization. How do zero-shot summarizers perform in other (potentially more specialized) domains? In this work we evaluate zero-shot generated summaries across specialized domains including biomedical articles, and legal bills (in addition to standard news benchmarks for reference). We focus especially on the factuality of outputs. We acquire annotations from domain experts to identify inconsistencies in summaries and systematically categorize these errors. We analyze whether the prevalence of a given domain in the pretraining corpus affects extractiveness and faithfulness of generated summaries of articles in this domain. We release all collected annotations to facilitate additional research toward measuring and realizing factually accurate summarization, beyond news articles. The dataset can be downloaded from https://github.com/sanjanaramprasad/zero_shot_faceval_domains
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
