To facilitate healthcare delivery, language models (LMs) have significant potential for clinical prediction tasks using electronic health records (EHRs). However, in these high-stakes applications, unreliable decisions can result in high costs due to compromised patient safety and ethical concerns, thus increasing the need for good uncertainty modeling of automated clinical predictions. To address this, we consider the uncertainty quantification of LMs for EHR tasks in white- and black-box settings. We first quantify uncertainty in white-box models, where we can access model parameters and output logits. We show that an effective reduction of model uncertainty can be achieved by using the proposed multi-tasking and ensemble methods in EHRs. Continuing with this idea, we extend our approach to black-box settings, including popular proprietary LMs such as GPT-4. We validate our framework using longitudinal clinical data from more than 6,000 patients in ten clinical prediction tasks. Results show that ensembling methods and multi-task prediction prompts reduce uncertainty across different scenarios. These findings increase the transparency of the model in white-box and black-box settings, thus advancing reliable AI healthcare.
HERMES: Contrast-Aware Knowledge Graph Reasoning from Clinical Notes for Patient Outcome Prediction
Business AI Lab, College of Technology, National Economics University（国民经济大学技术学院商业人工智能实验室）；Hanoi University of Science and Technology（河内理工大学）；A2I Lab, Phenikaa School of Computing, Phenikaa University（Phenikaa大学Phenikaa计算学院A2I实验室）；Department of Head and Neck Surgery, Vietnam National Cancer Hospital & Hanoi Medical University（越南国家癌症医院头颈外科与河内医科大学）；VNPT AI, VNPT Group（VNPT集团VNPT AI）
CT Open: An Open-Access, Uncontaminated, Live Platform for the Open Challenge of Clinical Trial Outcome Prediction
Laboratory for Emerging Intelligence, University of California, San Diego（新兴智能实验室，加州大学圣地亚哥分校）；Department of Dermatology, University of California, San Diego（皮肤科系，加州大学圣地亚哥分校）；Elsevier
DeepImagine: Clinical Trial Outcome Prediction via Stepwise Local Counterfactual Imaginations
Laboratory for Emerging Intelligence, University of California, San Diego（新兴智能实验室，加州大学圣地亚哥分校）；Department of Dermatology, University of California, San Diego（皮肤科系，加州大学圣地亚哥分校）；University of Chicago（芝加哥大学）；Elsevier（艾尔斯特出版社）
CovidLLM: A Robust Large Language Model with Missing Value Adaptation and Multi-Objective Learning Strategy for Predicting Disease Severity and Clinical Outcomes in COVID-19 Patients
School of Mathematics and Statistics, Xiamen University of Technology（厦门理工学院数学与统计学院）；School of Computer and Information Engineering, Xiamen University of Technology（厦门理工学院计算机与信息工程学院）；Tongji Medical College, Huazhong University of Science and Technology（华中科技大学同济医学院）
Beuth University of Applied Sciences Berlin（柏林比乌特应用科学大学）；Charité Berlin（柏林夏里特医学院）
Outcome prediction from clinical text can prevent doctors from overlooking possible risks and help hospitals to plan capacities. We simulate patients at admission time, when decision support can be especially valuable, and contribute a novel admission to discharge task with four common outcome prediction targets: Diagnoses at discharge, procedures performed, in-hospital mortality and length-of-stay prediction. The ideal system should infer outcomes based on symptoms, pre-conditions and risk factors of a patient. We evaluate the effectiveness of language models to handle this scenario and propo…
Imperial College London（帝国理工学院）；Pangaea Data Limited；Queen Mary University of London（伦敦玛丽女王大学）；Hong Kong Baptist University（香港浸会大学）
Objective: Clinical notes contain information not present elsewhere, including drug response and symptoms, all of which are highly important when predicting key outcomes in acute care patients. We propose the automatic annotation of phenotypes from clinical notes as a method to capture essential information, which is complementary to typically used vital signs and laboratory test results, to predict outcomes in the Intensive Care Unit (ICU). Methods: We develop a novel phenotype annotation model to annotate phenotypic features of patients which are then used as input features of predictive mod…
Automating the recognition of outcomes reported in clinical trials using machine learning has a huge potential of speeding up access to evidence necessary in healthcare decision-making. Prior research has however acknowledged inadequate training corpora as a challenge for the Outcome detection (OD) task. Additionally, several contextualized representations like BERT and ELMO have achieved unparalleled success in detecting various diseases, genes, proteins, and chemicals, however, the same cannot be emphatically stated for outcomes, because these models have been relatively under-tested and stu…
Georgia Institute of Technology（佐治亚理工学院）；Booz Allen Hamilton（博思艾伦汉密尔顿公司）
Due to increased access to clinical trial outcomes and analysis, researchers and scientists are able to iterate or improve upon relevant approaches more effectively. However, the metrics and related results of clinical trials typically do not follow any standardization in their reports, making it more difficult for researchers to parse the results of different trials. The objective of this paper is to describe an automated method utilizing natural language processing in order to describe the probable core outcomes of clinical trials, in order to alleviate the issues around disparate clinical t…
Carnegie Mellon University（卡内基梅隆大学）；Swedish Medical Group（瑞典医疗集团）；Allen Institute for Artificial Intelligence（艾伦人工智能研究所）；University of Washington（华盛顿大学）
We present BEEP (Biomedical Evidence-Enhanced Predictions), a novel approach for clinical outcome prediction that retrieves patient-specific medical literature and incorporates it into predictive models. Based on each individual patient's clinical notes, we train language models (LMs) to find relevant papers and fuse them with information from notes to predict outcomes such as in-hospital mortality. We develop methods to retrieve literature based on noisy, information-dense patient notes, and to augment existing outcome prediction models with retrieved papers in a manner that maximizes predict…
TOBB University of Economics and Technology（TOBB经济与技术大学）
Early prediction of mortality and length of stay(LOS) of a patient is vital for saving a patient's life and management of hospital resources. Availability of electronic health records(EHR) makes a huge impact on the healthcare domain and there has seen several works on predicting clinical problems. However, many studies did not benefit from the clinical notes because of the sparse, and high dimensional nature. In this work, we extract medical entities from clinical notes and use them as additional features besides time-series features to improve our predictions. We propose a convolution based …
