Despite being a unique source of information on patients' status and disease progression, clinical notes are characterized by high levels of duplication and information redundancy. In general domain text, it has been shown that deduplication does not harm language model (LM) pretraining, thus helping reduce the training cost. Although large LMs have proven to learn medical knowledge, they still require specialized domain adaptation for improved downstream clinical tasks. By leveraging large real-world clinical corpora, we first provided a fine-grained characterization of duplicates stemming from common writing practices and clinical relevancy. Second, we demonstrated that deduplicating clinical text can help clinical LMs encode less redundant information in a more efficient manner and do not harm classification tasks via prompt-based learning.
Robust Neural Stimulation Response Modeling Through Meta-Learning and Pretraining
University of Washington（华盛顿大学）；Paul G. Allen School of Computer Science & Engineering（保罗·G·艾伦计算机科学与工程学院）；Center for Neurotechnology（神经技术中心）；Computational Neuroscience Center（计算神经科学中心）；Department of Electrical and Computer Engineering（电气与计算机工程系）；Department of Bioengineering（生物工程系）
Boston Children’s Hospital（波士顿儿童医院）；Harvard Medical School（哈佛医学院）
Large scale contextual representation models, such as BERT, have significantly advanced natural language processing (NLP) in recently years. However, in certain area like healthcare, accessing diverse large scale text data from multiple institutions is extremely challenging due to privacy and regulatory reasons. In this article, we show that it is possible to both pretrain and fine tune BERT models in a federated manner using clinical texts from different silos without moving the data.
Detailed mobile sensing data from phones, watches, and fitness trackers offer an unparalleled opportunity to quantify and act upon previously unmeasurable behavioral changes in order to improve individual health and accelerate responses to emerging diseases. Unlike in natural language processing and computer vision, deep representation learning has yet to broadly impact this domain, in which the vast majority of research and clinical applications still rely on manually defined features and boosted tree models or even forgo predictive modeling altogether due to insufficient accuracy. This is du…
Clinical notes in electronic health records contain highly heterogeneous writing styles, including non-standard terminology or abbreviations. Using these notes in predictive modeling has traditionally required preprocessing (e.g. taking frequent terms or topic modeling) that removes much of the richness of the source data. We propose a pretrained hierarchical recurrent neural network model that parses minimally processed clinical notes in an intuitive fashion, and show that it improves performance for discharge diagnosis classification tasks on the Medical Information Mart for Intensive Care I…
Dementia-R1: Reinforced Pretraining and Reasoning from Unstructured Clinical Notes for Real-World Dementia Prognosis
Graduate School of AI, KAIST, Republic of Korea（韩国科学技术院人工智能研究生院）；Haeundae Paik Hospital, Inje University, Republic of Korea（韩国延世大学海云台医院）；Asan Medical Center, University of Ulsan College of Medicine, Republic of Korea（韩国釜山大学医学院阿桑医院）
Text Mining Unit Barcelona Supercomputing Center（文本挖掘组 巴塞罗那超级计算中心）
This work presents biomedical and clinical language models for Spanish by experimenting with different pretraining choices, such as masking at word and subword level, varying the vocabulary size and testing with domain data, looking for better language representations. Interestingly, in the absence of enough clinical data to train a model from scratch, we applied mixed-domain pretraining and cross-domain transfer approaches to generate a performant bio-clinical model suitable for real-world clinical data. We evaluated our models on Named Entity Recognition (NER) tasks for biomedical documents …
Hierarchical Pretraining for Biomedical Term Embeddings
Stanford University（斯坦福大学）；Tsinghua University（清华大学）；Beijing Institute of Technology（北京理工大学）；Alibaba Damo Academy（阿里巴巴达摩院）；Harvard T.H. Chan School of Public Health（哈佛陈曾熙公共卫生学院）
CardioEmbed: Domain-Specialized Text Embeddings for Clinical Cardiology
University of Nevada Las Vegas, Department of Neuroscience（内华达大学拉斯维加斯分校神经科学系）；Concorde Career Colleges, Department of Cardiovascular and Medical Diagnostic Sonography（康科德职业学院心血管与医学诊断超声系）
University of Washington（华盛顿大学）；George Mason University（乔治梅森大学）
Symptom information is primarily documented in free-text clinical notes and is not directly accessible for downstream applications. To address this challenge, information extraction approaches that can handle clinical language variation across different institutions and specialties are needed. In this paper, we present domain generalization for symptom extraction using pretraining and fine-tuning data that differs from the target domain in terms of institution and/or specialty and patient population. We extract symptom events using a transformer-based joint entity and relation extraction metho…
Clinical interactions are initially recorded and documented in free text medical notes. ICD coding is the task of classifying and coding all diagnoses, symptoms and procedures associated with a patient's visit. The process is often manual and extremely time-consuming and expensive for hospitals. In this paper, we propose a machine learning model, BERT-XML, for large scale automated ICD coding from EHR notes, utilizing recently developed unsupervised pretraining that have achieved state of the art performance on a variety of NLP tasks. We train a BERT model from scratch on EHR notes, learning w…
MIT（麻省理工学院）；Xyla（Xyla公司）；IBM Research（IBM研究院）；The Hospital for Sick Children（病童医院）；Brigham and Women’s Hospital（布莱根妇女医院）；Harvard Medical School（哈佛医学院）
Although recent advances in scaling large language models (LLMs) have resulted in improvements on many NLP tasks, it remains unclear whether these models trained primarily with general web text are the right tool in highly specialized, safety critical domains such as clinical text. Recent results have suggested that LLMs encode a surprising amount of medical knowledge. This raises an important question regarding the utility of smaller domain-specific language models. With the success of general-domain LLMs, is there still a need for specialized clinical models? To investigate this question, we…
