Pre-trained Large Language Models (LLMs) often struggle on out-of-domain datasets like healthcare focused text. We explore specialized pre-training to adapt smaller LLMs to different healthcare datasets. Three methods are assessed: traditional masked language modeling, Deep Contrastive Learning for Unsupervised Textual Representations (DeCLUTR), and a novel pre-training objective utilizing metadata categories from the healthcare settings. These schemes are evaluated on downstream document classification tasks for each dataset, with additional analysis of the resultant embedding spaces. Contrastively trained models outperform other approaches on the classification tasks, delivering strong performance from limited labeled data and with fewer model parameter updates required. While metadata-based pre-training does not further improve classifications across the datasets, it yields interesting embedding cluster separability. All domain adapted LLMs outperform their publicly available general base LLM, validating the importance of domain-specialization. This research illustrates efficient approaches to instill healthcare competency in compact LLMs even under tight computational budgets, an essential capability for responsible and sustainable deployment in local healthcare settings. We provide pre-training guidelines for specialized healthcare LLMs, motivate continued inquiry into contrastive objectives, and demonstrates adaptation techniques to align small LLMs with privacy-sensitive medical tasks.
Multimodal Deep Learning for Low-Resource Settings: A Vector Embedding Alignment Approach for Healthcare Applications
Massachusetts Institute of Technology(麻省理工学院);Universidad del Cauca(考卡大学);University of Michigan(密歇根大学);Harvard University(哈佛大学);University College Dublin(都柏林大学学院);São Paulo Federal University(圣保罗联邦大学);Harvard TH Chan School of Public Health(哈佛陈曾熙公共卫生学院);Beth Israel Deaconess Medical Center(贝斯以色列女执事医疗中心)
Developing and Evaluating an AI-Assisted Prediction Model for Unplanned Intensive Care Admissions following Elective Neurosurgery using Natural Language Processing within an Electronic Healthcare Record System
University College London(伦敦大学学院);National Hospital for Neurology and Neurosurgery(国立神经学与神经外科医院)
Empathy by Design: Aligning Large Language Models for Healthcare Dialogue
Department of Public Health Sciences, The University of Texas at El Paso, USA(公共卫生科学系,德克萨斯理工大学埃尔帕索分校);Department of Nursing, The University of Texas at El Paso, USA(护理系,德克萨斯理工大学埃尔帕索分校);Department of Rehabilitation Sciences, The University of Texas at El Paso, USA(康复科学系,德克萨斯理工大学埃尔帕索分校)
A Retrieval-Based Approach to Medical Procedure Matching in Romanian
National University of Science and Technology POLITEHNICA Bucharest(波兰技术大学科学与技术国家大学);Dalle Molle Institute for Artificial Intelligence Research (IDSIA)(达勒莫利人工智能研究所)
NLP for Maternal Healthcare: Perspectives and Guiding Principles in the Age of LLMs
Allen Institute for AI(艾伦人工智能研究所);Association of American Medical Colleges, Center for Health Justice(美国医学院协会健康正义中心);University of Washington(华盛顿大学);University of California, Berkeley(加州大学伯克利分校);University of California, San Francisco(加州大学旧金山分校)
Leveraging Large Language Models for Patient Engagement: The Power of Conversational AI in Digital Health
IBM T. J. Watson Research Center(IBM TJ沃森研究中心);Morehouse School of Medicine(莫尔豪斯医学院);Cleveland Clinic Foundation(克利夫兰诊所基金会);Queen’s University(女王大学);Red Hat(红帽公司)
Privacy Challenges and Solutions in Retrieval-Augmented Generation-Enhanced LLMs for Healthcare Chatbots: A Review of Applications, Risks, and Future Directions
Centre for Smart Health, School of Nursing(智能健康中心、护理学院);The Hong Kong Polytechnic University(香港理工大学);Department of Electrical and Electronic Engineering(电子与电气工程系);Faculty of Health Sciences(健康科学学院);University of Maribor(马里博大学)
MedBuild AI: An Agent-Based Hybrid Intelligence Framework for Reshaping Agency in Healthcare Infrastructure Planning through Generative Design for Medical Architecture
School of Architecture and Urban Planning, Beijing University of Civil Engineering and Architecture(北京建筑大学建筑与城市规划学院)
North Africa's Missing Framework: NLP-Driven Mental Healthcare in Algeria and Implications for Low-resource Settings
University Mohamed El Bachir El Ibrahimi of Bordj Bou Arreridj(布吉布阿雷里吉·穆罕默德·巴希尔·易卜拉欣大学);Laboratory of Informatics and its Applications of M’sila(姆西拉信息学及其应用实验室);Laboratory of Materials and Electronic Systems(材料与电子系统实验室)
VITAL: A New Dataset for Benchmarking Pluralistic Alignment in Healthcare
School of Computing and Information System, the University of Melbourne, Australia(墨尔本大学计算与信息学院);School of Computing, FSE, Macquarie University, Australia(麦考瑞大学计算学院)
ERVQA: A Dataset to Benchmark the Readiness of Large Vision Language Models in Hospital Environments
Indian Institute of Technology, Kharagpur(印度理工学院卡拉格普尔分校);All India Institute of Medical Sciences, Kalyani(卡里亚尼全印度医学科学研究所);All India Institute of Medical Sciences, Deoghar(迪奥加尔全印度医学科学研究所);Dr. B.C. Roy Multi-Speciality Medical Research Centre, IIT Kharagpur(印度理工学院卡拉格普尔分校Dr. B.C. Roy多专科医学研究中心)