Imperial College London(帝国理工学院);École des Ponts(巴黎路桥学院);EDF R&D(法国电力公司研发中心);Royal Brompton and Harefield Hospital(皇家布朗普顿和 Harefield 医院);Tongji University(同济大学)
In the field of medical Vision-Language Pre-training (VLP), significant efforts have been devoted to deriving text and image features from both clinical reports and associated medical images. However, most existing methods may have overlooked the opportunity in leveraging the inherent hierarchical structure of clinical reports, which are generally split into `findings' for descriptive content and `impressions' for conclusive observation. Instead of utilizing this rich, structured format, current medical VLP approaches often simplify the report into either a unified entity or fragmented tokens. In this work, we propose a novel clinical prior guided VLP framework named IMITATE to learn the structure information from medical reports with hierarchical vision-language alignment. The framework derives multi-level visual features from the chest X-ray (CXR) images and separately aligns these features with the descriptive and the conclusive text encoded in the hierarchical medical report. Furthermore, a new clinical-informed contrastive loss is introduced for cross-modal learning, which accounts for clinical prior knowledge in formulating sample correlations in contrastive learning. The proposed model, IMITATE, outperforms baseline VLP methods across six different datasets, spanning five medical imaging downstream tasks. Comprehensive experimental results highlight the advantages of integrating the hierarchical structure of medical reports for vision-language alignment. The code related to this paper is available at https://github.com/cheliu-computation/IMITATE-TMI2024.
Quantitative mapping from conventional MRI using self-supervised physics-guided deep learning: applications to a large-scale, clinically heterogeneous dataset
A WDLoRA-Based Multimodal Generative Framework for Clinically Guided Corneal Confocal Microscopy Image Synthesis in Diabetic Neuropathy
Department of Computing and Mathematics, Manchester Metropolitan University(曼彻斯特 Metropolitan 大学计算与数学系);Department of Eye and Vision Sciences, University of Liverpool(利物浦大学眼科学与视觉科学系);Faculty of Biology, Medicine and Health, University of Manchester(曼彻斯特大学生物、医学与健康学院);Department of Medicine, Weill Cornell Medicine-Qatar(韦尔·柯尔曼医学中心-卡塔尔医学系)
CLAIM: Clinically-Guided LGE Augmentation for Realistic and Diverse Myocardial Scar Synthesis and Segmentation
Insigneo Institute for in-silico, Medicine, University of Sheffield, Sheffield, UK(Insigneo医学研究所(in-silico),谢菲尔德大学);School of Computer Science, University of Sheffield, Sheffield, UK(谢菲尔德大学计算机科学学院);Department of Cardiology, Nottingham University Hospitals NHS Trust, Nottingham, UK(诺丁汉大学医院心脏科);School of Medicine, University of Nottingham, Nottingham, UK(诺丁汉大学医学院)
PD-Diag-Net: Clinical-Priors guided Network on Brain MRI for Auxiliary Diagnosis of Parkinson's Disease
Suzhou Institute for Advanced Research, University of Science and Technology of China(中国科学技术大学苏州市先进研究院);School of Artificial Intelligence and Data Science, University of Science and Technology of China(中国科学技术大学人工智能与数据科学学院);School of Electronic and Information Engineering, Beijing Jiaotong University(北京交通大学电子与信息工程学院);College of Control Science and Engineering, China University of Petroleum (East China)(中国石油大学(华东)控制科学与工程学院);School of Automation, Northwestern Polytechnical University(西北工业大学自动化学院);Beijing Tiantan Hospital(北京天坛医院)
Clinical Priors Guided Lung Disease Detection in 3D CT Scans
College of Computer Science and Artificial Intelligence, Shanghai Key Laboratory of Intelligent Information Processing, Fudan University(计算机科学与人工智能学院,上海智能信息处理重点实验室,复旦大学);University of Oxford(牛津大学);The Hong Kong University of Science and Technology(香港科技大学)
SGRNet: Spatially Guided Radiology Network for Structured Radiological Reporting of Head and Neck Cancer
University of Strasbourg(斯特拉斯堡大学);IHU Strasbourg(斯特拉斯堡大学医院研究所);Indian Institute of Technology (IIT) Madras(印度马德拉斯理工学院);All India Institute of Medical Sciences(全印度医学科学院)
Detecting Spurious Correlations with Sanity Tests for Artificial Intelligence Guided Radiology Systems
Memorial Sloan Kettering Cancer Center(纪念斯隆凯特琳癌症中心);Rochester Institute of Technology(罗切斯特理工学院);Institute of Research and Medical Care (IRCCS) SDN(研究与医疗护理研究所(IRCCS)SDN);University of Cagliari(卡利亚里大学)
Histopathological Spectrum-Guided Prostate Stratification via Segmentation-Assisted Diagnostic Transformer
College of Artificial Intelligence, Nankai University(南开大学人工智能学院);Tianjin First Central Hospital(天津市第一中心医院);School of Electronics and Information Engineering, Tiangong University(天津工业大学电子与信息工程学院);School of Artificial Intelligence, Hebei University of Technology(河北工业大学人工智能学院);North China Digital Health Technology Co., Ltd.(华北数字健康科技有限公司);School of Computing and Information Systems, Singapore Management University(新加坡管理大学计算与信息系统学院)
EG-SpikeFormer: Eye-Gaze Guided Transformer on Spiking Neural Networks for Medical Image Analysis
School of Computing, University of Georgia(佐治亚大学计算学院);University of Texas at Arlington(德克萨斯大学阿灵顿分校);Massachusetts General Hospital(麻省总医院);Harvard Medical School(哈佛医学院);University of Georgia(佐治亚大学);Department of Physics and Astronomy(物理与天文学系)
A Self-Guided Framework for Radiology Report Generation
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences(中国科学院深圳先进技术研究院);University of Chinese Academy of Sciences(中国科学院大学);Shenzhen University General Hospital(深圳大学总医院)
Clinically-Informed Modeling for Pediatric Brain Tumor Classification from Whole-Slide Histopathology Images
Dept. of Computer Science(计算机科学系);University of Texas at Austin(德克萨斯大学奥斯汀分校);School of Information(信息学院);Dell Children's Medical Center(德尔儿童医学中心);Dept. of OREI(OREI部门);University of Nevada, Reno(内华达大学里诺分校)
Aortic Valve Disease Screening from PPG via Physiology-Guided Self-Supervised Learning
Department of Computer Science, Tianjin University of Technology(天津理工大学计算机科学系);National Institute of Health Data Science, Peking University(北京大学国家健康数据科学研究院);Department of Cardiology, Peking University People’s Hospital(北京大学人民医院心内科);HeartVoice Medical Technology(心声医疗科技);DCST, BNRist, RIIT, Institute of Internet Industry, Tsinghua University(清华大学计算机科学与技术系、北京人工智能研究所、互联网产业研究所);Institute of Medical Technology, Peking University Health Science Center(北京大学健康科学中心医学技术研究所);Institute for Artificial Intelligence, Peking University(北京大学人工智能研究所);State Key Laboratory of Vascular Homeostasis and Remodeling, NHC Key Laboratory of Cardiovascular Molecular Biology and Regulatory Peptides, Peking University(北京大学血管稳态与重塑国家重点实验室、国家心血管分子生物学与调节肽重点实验室)
Self-Supervised Physics-Guided Deep Learning Reconstruction For High-Resolution 3D LGE CMR
University of Minnesota(明尼苏达大学);Center for Magnetic Resonance Research, University of Minnesota(明尼苏达大学磁共振研究中心);Beth Israel Deaconess Medical Center, Harvard Medical School(哈佛医学院贝斯以色列女执事医疗中心)
MR-CLIP: Efficient Metadata-Guided Learning of MRI Contrast Representations
School of Biomedical Engineering and Imaging Sciences, King’s College London, London, UK(生物医学工程与成像科学学院,伦敦国王学院,伦敦,英国);deepc GMBH, Munich, Germany(deepc 德国慕尼黑分公司)
Semantically Informed Salient Regions Guided Radiology Report Generation
School of Artificial Intelligence, Beijing University of Posts and Telecommunications(人工智能学院,北京邮电大学);Department of Radiology, Peking University Third Hospital(北京大学第三医院放射科)
Medical Knowledge-Guided Deep Learning for Imbalanced Medical Image Classification
University of Pittsburgh(匹兹堡大学);National University of Defense Technology(国防科技大学);University of Pittsburgh Medical Center Children’s Hospital of Pittsburgh(匹兹堡大学医学中心匹兹堡儿童医院);Magee-Womens Hospital of University of Pittsburgh Medical Center(匹兹堡大学医学中心马吉妇女医院)
Multimodal Doctor-in-the-Loop: A Clinically-Guided Explainable Framework for Predicting Pathological Response in Non-Small Cell Lung Cancer
Research Unit of Computer Systems and Bioinformatics, Department of Engineering(计算机系统与生物信息学研究单位,工程系);Research Unit of Radiation Oncology, Department of Medicine and Surgery(放射肿瘤学研究单位,医学与外科系);Research Unit of Anatomical Pathology, Department of Medicine and Surgery(病理学研究单位,医学与外科系)
IMITATE: Image Registration with Context for unknown time frame recovery
Centre de Recherches en Cancérologie de Toulouse, INSERM UMR1037, Team RADOPT(图卢兹癌症研究中心,INSERM UMR1037,RADOPT团队);Université de Toulouse(图卢兹大学);Institut Universitaire du Cancer – Oncopole Claudius Régaud(癌症大学研究所 – 克劳迪乌斯·雷加德肿瘤中心);Institut de Mathématiques de Toulouse (UMR 5219), CNRS, Université de Toulouse(图卢兹数学研究所(UMR 5219),CNRS,图卢兹大学)
Clinical-Prior Guided Multi-Modal Learning with Latent Attention Pooling for Gait-Based Scoliosis Screening
Orthopaedic Centre, The University of Hong Kong - Shenzhen Hospital, Shenzhen, China(香港大学深圳医院骨科中心);Translational Medicine Centre, The University of Hong Kong - Shenzhen Hospital, China(香港大学深圳医院转化医学中心);Department of Orthopaedics and Traumatology, Li Ka Shing Faculty of Medicine, The University of Hong Kong, Hong Kong, China(香港大学李嘉成医学院骨科及创伤外科学院);Huawei, China(华为)
Breast Ultrasound Tumor Generation via Mask Generator and Text-Guided Network:A Clinically Controllable Framework with Downstream Evaluation
School of Biomedical Engineering, Shenzhen University Medical School, Shenzhen University, Shenzhen, China(生物医学工程学院,深圳大学医学院,深圳大学,深圳,中国);Smart Medical Imaging, Learning and Engineering (SMILE) Lab, Shenzhen University, Shenzhen, China(智能医学影像、学习与工程(SMILE)实验室,深圳大学,深圳,中国);Nuclear Medicine Department, the Seventh Affiliated Hospital, Sun Yat-Sen University, Shenzhen, China(核医学科,中山大学第七附属医院,中山大学,深圳,中国);Department of Radiology, the Seventh Affiliated Hospital, Sun Yat-Sen University, Shenzhen, China(放射科,中山大学第七附属医院,中山大学,深圳,中国);Shenzhen University of Advanced Technology, Shenzhen, China(深圳大学先进技术大学,深圳,中国)
MGA: Medical generalist agent through text-guided knowledge transformation
Paul C. Lauterbur Research Center for Biomedical Imaging, Shenzhen Institutes of Advanced Technology(中国科学院深圳先进技术研究院保罗·C·劳特伯生物医学成像研究中心);University of Chinese Academy of Sciences(中国科学院大学);Pengcheng Laboratory(鹏城实验室);Renmin Hospital of Wuhan University(武汉大学人民医院)
REMEMBER: Retrieval-based Explainable Multimodal Evidence-guided Modeling for Brain Evaluation and Reasoning in Zero- and Few-shot Neurodegenerative Diagnosis
Lausanne University Hospital (CHUV)(洛桑大学医院(CHUV));University of Lausanne (UNIL)(洛桑大学(UNIL));Vietnam National University(越南国立大学);University of Science, Vietnam National University Ho Chi Minh City(越南国立大学胡志明市科学大学);International University, Vietnam National University Ho Chi Minh City(越南国立大学胡志明市国际大学)
Clinically-guided Data Synthesis for Laryngeal Lesion Detection
Biomedical Robotics Laboratory, Department of Advanced Robotics, Istituto Italiano di Tecnologia, Genoa, Italy Department of Informatics, Bioengineering, Robotics, Systems Engineering, University of Genova, Genoa, Italy Departament de Matem\`atiques i Inform\`atica, Universitat de Barcelona, Spain Computer Vision Center, Bellaterra, Spain Instituci\`o Catalana de Recerca i Estudis Avan cats (ICREA), Barcelona, Spain