FactPICO：医学证据通俗语言摘要的事实性评估
FactPICO: Factuality Evaluation for Plain Language Summarization of Medical Evidence
AI总结：
本文提出FactPICO，一个针对医学RCT文本通俗语言摘要的事实性评估基准，包含345篇由多个LLM生成的摘要及专家评估，研究发现现有指标与专家判断相关性差，平衡简洁与事实性仍具挑战。
摘要：
使用大语言模型（LLM）进行通俗语言摘要有助于提高技术内容的文本可及性。但在医学等高风险领域，这些摘要的事实性如何？本文提出了FactPICO，一个用于医学文本通俗语言摘要的事实性基准，这些文本描述了随机对照试验（RCT），后者是循证医学的基础并能直接指导患者治疗。FactPICO包含345篇由三个LLM（即GPT-4、Llama-2和Alpaca）生成的RCT摘要通俗语言摘要，并附带专家提供的细粒度评估和自然语言理由。我们评估了这些摘要中RCT关键要素的事实性：人群、干预措施、对照、结局（PICO），以及与这些要素相关的报告发现。我们还评估了LLM添加的额外信息（例如解释）的正确性。利用FactPICO，我们对一系列现有的事实性指标进行了基准测试，包括基于LLM新设计的指标。我们发现，医学证据的通俗语言摘要仍然具有挑战性，尤其是在平衡简洁性与事实性时，并且现有指标在实例级别上与专家判断的相关性很差。
英文摘要：
Plain language summarization with LLMs can be useful for improving textual accessibility of technical content. But how factual are these summaries in a high-stakes domain like medicine? This paper presents FactPICO, a factuality benchmark for plain language summarization of medical texts describing randomized controlled trials (RCTs), which are the basis of evidence-based medicine and can directly inform patient treatment. FactPICO consists of 345 plain language summaries of RCT abstracts generated from three LLMs (i.e., GPT-4, Llama-2, and Alpaca), with fine-grained evaluation and natural language rationales from experts. We assess the factuality of critical elements of RCTs in those summaries: Populations, Interventions, Comparators, Outcomes (PICO), as well as the reported findings concerning these. We also evaluate the correctness of the extra information (e.g., explanations) added by LLMs. Using FactPICO, we benchmark a range of existing factuality metrics, including the newly devised ones based on LLMs. We find that plain language summarization of medical evidence is still challenging, especially when balancing between simplicity and factuality, and that existing metrics correlate poorly with expert judgments on the instance level.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
