Development and Testing of Retrieval Augmented Generation in Large Language Models -- A Case Study Report
Singapore General Hospital(新加坡中央医院);Duke-NUS Medical School(杜克-新加坡国立大学医学院);Singapore National Eye Centre(新加坡全国眼科中心);Singapore Eye Research Institute(新加坡眼科研究所);Singapore Health Services(新加坡保健服务)
YuHe Ke, Liyuan Jin, Kabilan Elangovan, Hairil Rizal Abdullah, Nan Liu, Alex Tiong Heng Sia, Chai Rick Soh, Joshua Yi Min Tung, Jasmine Chiat Ling Ong, Daniel Shu Wei Ting
Purpose: Large Language Models (LLMs) hold significant promise for medical applications. Retrieval Augmented Generation (RAG) emerges as a promising approach for customizing domain knowledge in LLMs. This case study presents the development and evaluation of an LLM-RAG pipeline tailored for healthcare, focusing specifically on preoperative medicine.
Methods: We developed an LLM-RAG model using 35 preoperative guidelines and tested it against human-generated responses, with a total of 1260 responses evaluated. The RAG process involved converting clinical documents into text using Python-based frameworks like LangChain and Llamaindex, and processing these texts into chunks for embedding and retrieval. Vector storage techniques and selected embedding models to optimize data retrieval, using Pinecone for vector storage with a dimensionality of 1536 and cosine similarity for loss metrics. Human-generated answers, provided by junior doctors, were used as a comparison.
Results: The LLM-RAG model generated answers within an average of 15-20 seconds, significantly faster than the 10 minutes typically required by humans. Among the basic LLMs, GPT4.0 exhibited the best accuracy of 80.1%. This accuracy was further increased to 91.4% when the model was enhanced with RAG. Compared to the human-generated instructions, which had an accuracy of 86.3%, the performance of the GPT4.0 RAG model demonstrated non-inferiority (p=0.610).
Conclusions: In this case study, we demonstrated a LLM-RAG model for healthcare implementation. The pipeline shows the advantages of grounded knowledge, upgradability, and scalability as important aspects of healthcare LLM deployment.
Beyond the Parameters: A Technical Survey of Contextual Enrichment in Large Language Models: From In-Context Prompting to Causal Retrieval-Augmented Generation
Benchmarking Foundation Models with Retrieval-Augmented Generation in Olympic-Level Physics Problem Solving
AAII, University of Technology Sydney, New South Wales, Australia(AAII,悉尼大学,新南威尔士州,澳大利亚);Eindhoven University of Technology, Eindhoven, The Netherlands(埃因霍温理工大学,埃因霍温,荷兰);University of Liverpool, Liverpool, United Kingdom(利物浦大学,利物浦,英国);University of New South Wales, New South Wales, Australia(新南威尔士大学,新南威尔士州,澳大利亚)
Improving Context Fidelity via Native Retrieval-Augmented Reasoning
DIRO, Université de Montréal(蒙特利尔大学计算机科学与运筹学系);MetaGPT;Mila - Quebec AI Institute(米拉-魁北克人工智能研究所);McGill University(麦吉尔大学);Yale University(耶鲁大学);Canada CIFAR AI Chair(加拿大CIFAR人工智能教席)
A Systematic Study of Retrieval Pipeline Design for Retrieval-Augmented Medical Question Answering
Department of Mechatronics & Industrial Engineering, Chittagong University of Engineering & Technology(吉大港工程与技术大学机电与工业工程系);Department of Mechanical Engineering, Chittagong University of Engineering & Technology(吉大港工程与技术大学机械工程系)
Towards Comprehensive Vietnamese Retrieval-Augmented Generation and Large Language Models
Foundation Models Lab, BKAI(BKAI基础模型实验室);Hanoi University of Science and Technology(河内科技大学);University of Information Technology(信息技术大学);Vietnam National University HCMC(胡志明市越南国家大学)
R^2AG: Incorporating Retrieval Information into Retrieval Augmented Generation
School of Computer Science, South China Normal University(华南师范大学计算机学院);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));The Hong Kong University of Science and Technology(香港科技大学)
Retrieval-Augmented Large Language Models for Evidence-Informed Guidance on Cannabidiol Use in Older Adults
Lawrence Bloomberg Faculty of Nursing, University of Toronto(多伦多大学劳伦斯·布隆伯格护理学院);KITE Research Institute, Toronto Rehabilitation Institute, University Health Network(大学健康网络多伦多康复研究所KITE研究所);College of Engineering and Technology, American University of the Middle East(中东美国大学工程与技术学院)