arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2305.06299cs.CL
收藏

使用GPT-3总结、简化与综合医学证据(成效不一)

Summarizing, Simplifying, and Synthesizing Medical Evidence Using GPT-3 (with Varying Success)

Northeastern University(东北大学);The University of Texas at Austin(德克萨斯大学奥斯汀分校);King’s College London(伦敦国王学院)

Chantal Shaib, Millicent L. Li, Sebastian Joseph, Iain J. Marshall, Junyi Jessy Li, Byron C. Wallace

AI总结:

本文评估GPT-3在生物医学领域零样本生成摘要的能力,发现其能忠实总结单篇文献,但综合多文档证据时准确性不足。

摘要:

大型语言模型,特别是GPT-3,能够在少样本和零样本设置下生成通用领域新闻文章的高质量摘要。然而,尚不清楚此类模型在生物医学等更专业、高风险领域是否同样具备能力。在本文中,我们邀请领域专家(具有医学培训背景的个人)在零监督条件下评估GPT-3生成的生物医学文章摘要。我们考虑了单文档和多文档两种设置。在前者中,GPT-3的任务是生成描述随机对照试验文章的常规摘要和通俗语言摘要;在后者中,我们评估GPT-3在多大程度上能够综合一组文章中报告的证据。我们设计了一个用于评估模型输出的标注方案,重点评估生成摘要的事实准确性。我们发现,尽管GPT-3能够忠实地总结和简化单篇生物医学文章,但在多文档上提供准确的发现聚合方面存在困难。我们发布了本工作中使用的所有数据和标注。

英文摘要:

Large language models, particularly GPT-3, are able to produce high quality summaries of general domain news articles in few- and zero-shot settings. However, it is unclear if such models are similarly capable in more specialized, high-stakes domains such as biomedicine. In this paper, we enlist domain experts (individuals with medical training) to evaluate summaries of biomedical articles generated by GPT-3, given zero supervision. We consider both single- and multi-document settings. In the former, GPT-3 is tasked with generating regular and plain-language summaries of articles describing randomized controlled trials; in the latter, we assess the degree to which GPT-3 is able to \emph{synthesize} evidence reported across a collection of articles. We design an annotation scheme for evaluating model outputs, with an emphasis on assessing the factual accuracy of generated summaries. We find that while GPT-3 is able to summarize and simplify single biomedical articles faithfully, it struggles to provide accurate aggregations of findings over multiple documents. We release all data and annotations used in this work.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑