Lightweight Transformers for Clinical Natural Language Processing
University of Oxford（牛津大学）；NLPie Research（NLPie研究院）；University of Surrey（萨里大学）；Oxford-Suzhou Centre for Advanced Research（牛津苏州高等研究院）；Nuffield Department of Population Health, University of Oxford（牛津大学纳菲尔德人口健康系）；Pandemic Sciences Institute, University of Oxford（牛津大学流行病科学研究所）；Sharif University of Technology（谢里夫理工大学）；Infectious Diseases Data Observatory (IDDO), University of Oxford（牛津大学传染病数据观测站（IDDO））
Omid Rohanian, Mohammadmahdi Nouriborji, Hannah Jauncey, Samaneh Kouchaki, ISARIC Clinical Characterisation Group, Lei Clifton, Laura Merson, David A. Clifton
Specialised pre-trained language models are becoming more frequent in NLP since they can potentially outperform models trained on generic texts. BioBERT and BioClinicalBERT are two examples of such models that have shown promise in medical NLP tasks. Many of these models are overparametrised and resource-intensive, but thanks to techniques like Knowledge Distillation (KD), it is possible to create smaller versions that perform almost as well as their larger counterparts. In this work, we specifically focus on development of compact language models for processing clinical texts (i.e. progress notes, discharge summaries etc). We developed a number of efficient lightweight clinical transformers using knowledge distillation and continual learning, with the number of parameters ranging from 15 million to 65 million. These models performed comparably to larger models such as BioBERT and ClinicalBioBERT and significantly outperformed other compact models trained on general or biomedical data. Our extensive evaluation was done across several standard datasets and covered a wide range of clinical text-mining tasks, including Natural Language Inference, Relation Extraction, Named Entity Recognition, and Sequence Classification. To our knowledge, this is the first comprehensive study specifically focused on creating efficient and compact transformers for clinical NLP tasks. The models and code used in this study can be found on our Huggingface profile at https://huggingface.co/nlpie and Github page at https://github.com/nlpie-research/Lightweight-Clinical-Transformers, respectively, promoting reproducibility of our results.
Lightweight Retrieval-Augmented Generation and Large Language Model-Based Modeling for Scalable Patient-Trial Matching
Department of Artificial Intelligence and Informatics, Mayo Clinic（人工智能与信息学系，梅奥诊所）；Computer Science Department, University of Tulsa（图兰大学计算机科学系）；Mayo Clinic Comprehensive Cancer Center, Mayo Clinic（梅奥诊所综合癌症中心，梅奥诊所）；Division of Community Pediatric and Adolescent Medicine, Department of Pediatrics, Mayo Clinic（社区儿科与青少年医学分会，儿科部，梅奥诊所）；Department of Neurology, Mayo Clinic（神经病学部，梅奥诊所）；Department of Radiation Oncology, Mayo Clinic（放射肿瘤学部，梅奥诊所）；Department of Quantitative Health Sciences, Mayo Clinic（定量健康科学部，梅奥诊所）
A Monosemantic Attribution Framework for Stable Interpretability in Clinical Neuroscience Transformer-Based Language Models
Department of Computer Science and Technology（计算机科学与技术系）；Cancer Research UK（癌症研究英国公司）；Cambridge Institute（剑桥研究所）；University of Cambridge（剑桥大学）；United Kingdom（英国）；DIMES（迪梅斯）；University of Calabria（卡拉布里亚大学）；Italy（意大利）；Department of Computer Automatic and Management Engineering (DIAG)（计算机自动与管理工程系）；Sapienza Università di Roma（罗马大学萨皮恩扎）；Department of Psychiatry（精神病学系）；School of Computing（计算学院）；University of Kent（肯特大学）；Department of Psychology（心理学系）
SkinDistilViT: Lightweight Vision Transformer for Skin Lesion Classification
National Institute for Research and Development in Informatics - ICI Bucharest（布加勒斯特国家信息学研发研究院（ICI布加勒斯特））；Faculty of Automatic Control and Computers, University Politehnica of Bucharest（布加勒斯特理工大学自动控制与计算机学院）
本文提出 SkinDistilViT，通过分层分类头与级联知识蒸馏，将专家标注图像训练的皮肤病变 ViT 压缩为低成本轻量模型，并在准确率接近教师模型的同时显著降低 GPU/CPU 推理开销。
Classifying Long Clinical Documents with Pre-trained Transformers
University of Arizona（亚利桑那大学）；Boston Children’s Hospital（波士顿儿童医院）；Harvard Medical School（哈佛医学院）；University of Wisconsin–Madison（威斯康星大学麦迪逊分校）；Loyola University Chicago（芝加哥洛约拉大学）
Hierarchical Label-wise Attention Transformer Model for Explainable ICD Coding
University of New South Wales（新南威尔士大学）；The Australian e-Health Research Centre, CSIRO（澳大利亚联邦科学与工业研究组织澳大利亚电子健康研究中心）；Australian Institute of Health and Welfare（澳大利亚卫生与福利研究所）
Extracting Angina Symptoms from Clinical Notes Using Pre-Trained Transformer Architectures
Brown University（布朗大学）；The Warren Alpert Medical School, Brown University（布朗大学沃伦·阿尔珀特医学院）；School of Public Health, Brown University（布朗大学公共卫生学院）；Providence VA Medical Center（普罗维登斯退伍军人事务医疗中心）；Rhode Island Quality Institute（罗德岛质量研究所）
Clinical Context-aware Radiology Report Generation from Medical Images using Transformers
School of Computer Science and Engineering（计算机科学与工程学院）；Faculty of Engineering, University of New South Wales（新南威尔士大学工程学院）；School of Computing（计算机学院）；Faculty of Science and Engineering, Macquarie University（麦考瑞大学科学与工程学院）
本研究提出基于 transformer 的架构从胸部 X 光片生成放射学报告，实验证明其优于 LSTM 且速度更快，并强调需结合语言生成与分类指标以全面评估报告的连贯性与诊断价值。
AD-CDO: A Lightweight Ontology for Representing Eligibility Criteria in Alzheimer's Disease Clinical Trials
University of Texas Health Science Center at Houston（德克萨斯大学休斯顿健康科学中心）；Indiana University School of Medicine（印第安纳大学医学院）；Mayo Clinic in Florida（佛罗里达梅奥诊所）
A Transformer-based representation-learning model with unified processing of multimodal input for clinical diagnostics
The University of Hong Kong（香港大学）；West China Hospital, Sichuan University（四川大学华西医院）；Deepwise Healthcare（深睿医疗）；Zhuhai People’s Hospital and the First Affiliated Hospital of Faculty of Medicine, Macau University of Science and Technology and University Hospital（澳门科技大学医学院附属第一医院、珠海市人民医院及大学医院）；Peking University（北京大学）；Guangzhou Laboratory（广州实验室）；Jinling Hospital, Nanjing University School of Medicine（南京大学医学院附属金陵医院）
Re-Parameterization of Lightweight Transformer for On-Device Speech Emotion Recognition
College of Computer Science and Electronic Engineering, Hunan University（湖南大学计算机与电子工程学院）；Shenzhen Research Institute, Hunan University（湖南大学深圳研究院）；University of Cambridge（剑桥大学）
Discharge Summary Hospital Course Summarisation of In Patient Electronic Health Record Text with Clinical Concept Guided Deep Pre-Trained Transformer Models
King’s College London（伦敦国王学院）；King’s College Hospital NHS Foundation Trust（国王学院医院NHS基金会信托）；University College London（伦敦大学学院）
