Parameter-Efficient Fine-Tuning (PEFT) and Retrieval-Augmented Generation (RAG) have become popular methods for adapting large language models while minimizing compute requirements. In this paper, we apply PEFT methods (P-tuning, Adapters, and LoRA) to a modified Retrieval-Enhanced Transformer (RETRO) and a baseline GPT model across several sizes, ranging from 823 million to 48 billion parameters. We show that RETRO models outperform GPT models in zero-shot settings due to their unique pre-training process but GPT models have higher performance potential with PEFT. Additionally, our study indicates that 8B parameter models strike an optimal balance between cost and performance and P-tuning lags behind other PEFT techniques. We further provide a comparative analysis between applying PEFT to an Instruction-tuned RETRO model and base RETRO model. This work presents the first comprehensive comparison of various PEFT methods integrated with RAG, applied to both GPT and RETRO models, highlighting their relative performance.
The capabilities of natural language models trained on large-scale data have increased immensely over the past few years. Open source libraries such as HuggingFace have made these models easily available and accessible. While prior research has identified biases in large language models, this paper considers biases contained in the most popular versions of these models when applied `out-of-the-box' for downstream tasks. We focus on generative language models as they are well-suited for extracting biases inherited from training data. Specifically, we conduct an in-depth analysis of GPT-2, which…
We enhance auto-regressive language models by conditioning on document chunks retrieved from a large corpus, based on local similarity with preceding tokens. With a $2$ trillion token database, our Retrieval-Enhanced Transformer (RETRO) obtains comparable performance to GPT-3 and Jurassic-1 on the Pile, despite using 25$\times$ fewer parameters. After fine-tuning, RETRO performance translates to downstream knowledge-intensive tasks such as question answering. RETRO combines a frozen Bert retriever, a differentiable encoder and a chunked cross-attention mechanism to predict tokens based on an o…
Large Language Model-Based Uncertainty-Adjusted Label Extraction for Artificial Intelligence Model Development in Upper Extremity Radiography
Lab for Artificial Intelligence in Medicine, Department of Diagnostic and Interventional Radiology, University Hospital Aachen（人工智能医学实验室，诊断与介入放射科，亚琛大学医院）；Department of Diagnostic and Interventional Radiology, University Hospital Aachen（诊断与介入放射科，亚琛大学医院）；Institute for Diagnostic and Interventional Radiology, Faculty of Medicine and University Hospital Cologne, University of Cologne（诊断与介入放射学研究所，医学学院和科隆大学医院，科隆大学）；Department of Diagnostic and Interventional Radiology, University Medical Center Mainz（诊断与介入放射科，马因茨大学医学中心）
Emergency Department Revisit Quality Review Screening: Exploring Human Decision-Making and Artificial Intelligence Support
OSF HealthCare（OSF医疗保健）；University of Illinois College of Medicine at Peoria（伊利诺伊大学皮奥里亚医学院）；Saint Anthony College of Nursing（圣安东尼护理学院）；Thomas Jefferson University's Sidney Kimmel College of Medicine（托马斯·杰斐逊大学西德尼·金梅尔医学院）；Keylog Solutions LLC（Keylog解决方案有限责任公司）
EHR-MCP: Real-world Evaluation of Clinical Information Retrieval by Large Language Models via Model Context Protocol
Department of Radiology, Keio University Hospital, Tokyo, Japan（Keio大学医院放射科）；Division of Infectious Diseases and Infection Control, Keio University Hospital, Tokyo, Japan（Keio大学医院感染病学与感染控制科）
Language models have become increasingly popular in recent years for tasks like information retrieval. As use-cases become oriented toward specific domains, fine-tuning becomes default for standard performance. To fine-tune these models for specific tasks and datasets, it is necessary to carefully tune the model's hyperparameters and training techniques. In this paper, we present an in-depth analysis of the performance of four transformer-based language models on the task of biomedical information retrieval. The models we consider are DeepMind's RETRO (7B parameters), GPT-J (6B parameters), GP…
