面向医学的检索增强生成基准评测
Benchmarking Retrieval-Augmented Generation for Medicine
AI总结：
本文提出医学RAG评测基准MIRAGE和MedRAG工具包，通过大规模实验系统评估不同语料库、检索器与LLM组合，发现最佳配置可将模型准确率提升至多18%，并为医学RAG实践提供指导。
摘要：
尽管大语言模型（LLMs）在广泛的医学问答（QA）任务上已取得最先进的性能，它们仍面临幻觉和知识过时的挑战。检索增强生成（RAG）是一种有前景的解决方案，并已被广泛采用。然而，一个RAG系统可能涉及多个灵活组件，针对不同医学目的的最优RAG设置仍缺乏最佳实践。为了系统性地评估此类系统，我们提出了医学信息检索增强生成评测基准（MIRAGE），这是首个此类基准，包含来自五个医学QA数据集的7,663个问题。利用MIRAGE，我们通过本工作引入的MedRAG工具包，在41种不同语料库、检索器和骨干LLM的组合上进行了超过1.8万亿提示token的大规模实验。总体而言，MedRAG将六种不同LLM的准确率相比思维链提示提高了最多18%，使GPT-3.5和Mixtral的性能提升至GPT-4水平。我们的结果表明，多种医学语料库与检索器的组合取得了最佳性能。此外，我们发现了医学RAG中的对数线性扩展特性和“迷失在中间”效应。我们相信，我们的全面评测可以作为在医学领域实施RAG系统的实用指南。
英文摘要：
While large language models (LLMs) have achieved state-of-the-art performance on a wide range of medical question answering (QA) tasks, they still face challenges with hallucinations and outdated knowledge. Retrieval-augmented generation (RAG) is a promising solution and has been widely adopted. However, a RAG system can involve multiple flexible components, and there is a lack of best practices regarding the optimal RAG setting for various medical purposes. To systematically evaluate such systems, we propose the Medical Information Retrieval-Augmented Generation Evaluation (MIRAGE), a first-of-its-kind benchmark including 7,663 questions from five medical QA datasets. Using MIRAGE, we conducted large-scale experiments with over 1.8 trillion prompt tokens on 41 combinations of different corpora, retrievers, and backbone LLMs through the MedRAG toolkit introduced in this work. Overall, MedRAG improves the accuracy of six different LLMs by up to 18% over chain-of-thought prompting, elevating the performance of GPT-3.5 and Mixtral to GPT-4-level. Our results show that the combination of various medical corpora and retrievers achieves the best performance. In addition, we discovered a log-linear scaling property and the "lost-in-the-middle" effects in medical RAG. We believe our comprehensive evaluations can serve as practical guidelines for implementing RAG systems for medicine.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
