用于临床摘要事实对齐的合成模仿编辑反馈
Synthetic Imitation Edit Feedback for Factual Alignment in Clinical Summarization
AI总结：
针对临床摘要任务中大语言模型易出现事实幻觉、人工标注反馈成本高的问题，提出用ChatGPT生成专家级编辑反馈的新流水线，验证了其在事实对齐方面的应用潜力。
摘要：
GPT、LLaMA系列等大语言模型（LLM）已展现出捕捉和浓缩关键上下文信息的卓越能力，在摘要任务中取得了当前最优性能。然而，业界对这些模型幻觉问题的担忧持续加剧。LLM有时会生成存在事实幻觉的摘要，这在临床领域自然语言处理（NLP）任务（如临床病历摘要）中危害极大，因为事实性错误表述可能导致严重的诊断失误。利用人类反馈微调LLM已被证明有望让模型在生成时保持事实一致性，但这类训练流程需要高质量的人工标注数据，而在临床领域获取这类数据的成本极高。本研究提出一种新流水线，用ChatGPT替代人类专家生成高质量反馈数据，以提升临床病历摘要任务中的事实一致性。我们特别聚焦于编辑反馈，因为近期研究指出，在复杂场景（如需要大量专业知识的临床NLP任务）中，基于偏好反馈的人类对齐存在不足，而从领域专家处收集编辑反馈具备一定优势。此外，尽管GPT已在多项临床NLP任务（如美国医师执照考试问答（USMLE QA））中达到专家水平，但此前鲜有研究探讨GPT能否在临床病历摘要任务中为语言模型生成专家级编辑反馈，我们希望填补这一空白。最终，我们的评估证明了GPT编辑在人类对齐中的应用潜力，尤其是在事实性维度。
英文摘要：
Large Language Models (LLMs) like the GPT and LLaMA families have demonstrated exceptional capabilities in capturing and condensing critical contextual information and achieving state-of-the-art performance in the summarization task. However, community concerns about these models' hallucination issues continue to rise. LLMs sometimes generate factually hallucinated summaries, which can be extremely harmful in the clinical domain NLP tasks (e.g., clinical note summarization), where factually incorrect statements can lead to critically erroneous diagnoses. Fine-tuning LLMs using human feedback has shown the promise of aligning LLMs to be factually consistent during generation, but such training procedure requires high-quality human-annotated data, which can be extremely expensive to get in the clinical domain. In this work, we propose a new pipeline using ChatGPT instead of human experts to generate high-quality feedback data for improving factual consistency in the clinical note summarization task. We focus specifically on edit feedback because recent work discusses the shortcomings of human alignment via preference feedback in complex situations (such as clinical NLP tasks that require extensive expert knowledge), as well as some advantages of collecting edit feedback from domain experts. In addition, although GPT has reached the expert level in many clinical NLP tasks (e.g., USMLE QA), there is not much previous work discussing whether GPT can generate expert-level edit feedback for LMs in the clinical note summarization task. We hope to fill this gap. Finally, our evaluations demonstrate the potential use of GPT edits in human alignment, especially from a factuality perspective.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
