K-QA：一个真实世界医疗问答基准
K-QA: A Real-World Medical Q&A Benchmark
AI总结：
该研究构建了包含1212条真实患者问题的医疗问答基准K-QA，提出基于NLI的全面性和幻觉率评估指标，验证了上下文学习可提升模型回复全面性、增强检索能减少幻觉，为医疗准确型NLP研究提供支持。
摘要：
确保大语言模型（LLM）生成回复的准确性至关重要，在临床场景中错误信息可能直接影响患者健康的情况下尤其如此。为应对这一挑战，我们构建了K-QA数据集，包含1212条来自K Health（一个AI驱动的临床平台）真实对话的患者问题。我们组建了内部医师团队来回答问题，并将K-QA的一个子集手动拆解为独立陈述。此外，我们提出了两个基于自然语言推理（NLI）的评估指标，用于近似衡量召回率和精确率：（1）全面性，衡量生成答案中关键临床信息的占比；（2）幻觉率，衡量医师整理的回复中被LLM答案矛盾的陈述数量。最后，我们使用K-QA及上述指标评估了多个前沿模型，以及上下文学习和作者开发的医疗导向增强检索方案的效果。研究结果表明，上下文学习提升了模型的全面性，增强检索可有效减少幻觉。我们将K-QA向社区开放，以推动医疗准确型自然语言处理应用的研究。
英文摘要：
Ensuring the accuracy of responses provided by large language models (LLMs) is crucial, particularly in clinical settings where incorrect information may directly impact patient health. To address this challenge, we construct K-QA, a dataset containing 1,212 patient questions originating from real-world conversations held on K Health (an AI-driven clinical platform). We employ a panel of in-house physicians to answer and manually decompose a subset of K-QA into self-contained statements. Additionally, we formulate two NLI-based evaluation metrics approximating recall and precision: (1) comprehensiveness, measuring the percentage of essential clinical information in the generated answer and (2) hallucination rate, measuring the number of statements from the physician-curated response contradicted by the LLM answer. Finally, we use K-QA along with these metrics to evaluate several state-of-the-art models, as well as the effect of in-context learning and medically-oriented augmented retrieval schemes developed by the authors. Our findings indicate that in-context learning improves the comprehensiveness of the models, and augmented retrieval is effective in reducing hallucinations. We make K-QA available to to the community to spur research into medically accurate NLP applications.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
