arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.01733cs.CLcs.AI
收藏

大型语言模型中检索增强生成(RAG)的开发与测试——案例研究报告

Development and Testing of Retrieval Augmented Generation in Large Language Models -- A Case Study Report

Singapore General Hospital(新加坡中央医院);Duke-NUS Medical School(杜克-新加坡国立大学医学院);Singapore National Eye Centre(新加坡全国眼科中心);Singapore Eye Research Institute(新加坡眼科研究所);Singapore Health Services(新加坡保健服务)

YuHe Ke, Liyuan Jin, Kabilan Elangovan, Hairil Rizal Abdullah, Nan Liu, Alex Tiong Heng Sia, Chai Rick Soh, Joshua Yi Min Tung, Jasmine Chiat Ling Ong, Daniel Shu Wei Ting

AI总结:

本研究开发并测试了用于术前医学的LLM-RAG流程,使用35份指南和1260个回答,GPT4.0结合RAG达到91.4%准确率,优于人工的86.3%,且生成速度显著更快。

摘要:

目的:大型语言模型(LLMs)在医学应用中具有巨大潜力。检索增强生成(RAG)作为一种在LLMs中定制领域知识的有前景的方法而出现。本案例研究介绍了一个专为医疗保健(特别是术前医学)定制的LLM-RAG流程的开发与评估。方法:我们使用35份术前指南开发了一个LLM-RAG模型,并将其与人工生成的回答进行了对比测试,共评估了1260个回答。RAG流程包括使用基于Python的框架(如LangChain和Llamaindex)将临床文档转换为文本,并将这些文本处理成块(chunks)以进行嵌入和检索。我们采用向量存储技术和选定的嵌入模型来优化数据检索,使用Pinecone进行向量存储,维度为1536,并使用余弦相似度作为损失度量。由初级医生提供的人工生成回答作为比较基准。结果:LLM-RAG模型生成回答的平均时间为15-20秒,显著快于人类通常所需的10分钟。在基础LLM中,GPT4.0表现出最佳准确率,为80.1%。当模型通过RAG增强后,该准确率进一步提高到91.4%。与人工生成的指令(准确率为86.3%)相比,GPT4.0 RAG模型的性能显示出非劣效性(p=0.610)。结论:在本案例研究中,我们展示了一个用于医疗保健实施的LLM-RAG模型。该流程展示了基于知识的优势、可升级性和可扩展性,这些是医疗保健LLM部署的重要方面。

英文摘要:

Purpose: Large Language Models (LLMs) hold significant promise for medical applications. Retrieval Augmented Generation (RAG) emerges as a promising approach for customizing domain knowledge in LLMs. This case study presents the development and evaluation of an LLM-RAG pipeline tailored for healthcare, focusing specifically on preoperative medicine. Methods: We developed an LLM-RAG model using 35 preoperative guidelines and tested it against human-generated responses, with a total of 1260 responses evaluated. The RAG process involved converting clinical documents into text using Python-based frameworks like LangChain and Llamaindex, and processing these texts into chunks for embedding and retrieval. Vector storage techniques and selected embedding models to optimize data retrieval, using Pinecone for vector storage with a dimensionality of 1536 and cosine similarity for loss metrics. Human-generated answers, provided by junior doctors, were used as a comparison. Results: The LLM-RAG model generated answers within an average of 15-20 seconds, significantly faster than the 10 minutes typically required by humans. Among the basic LLMs, GPT4.0 exhibited the best accuracy of 80.1%. This accuracy was further increased to 91.4% when the model was enhanced with RAG. Compared to the human-generated instructions, which had an accuracy of 86.3%, the performance of the GPT4.0 RAG model demonstrated non-inferiority (p=0.610). Conclusions: In this case study, we demonstrated a LLM-RAG model for healthcare implementation. The pipeline shows the advantages of grounded knowledge, upgradability, and scalability as important aspects of healthcare LLM deployment.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑