Medical Image Synthesis via Fine-Grained Image-Text Alignment and Anatomy-Pathology Prompting
City University of Hong Kong（香港城市大学）；The Chinese University of Hong Kong（香港中文大学）；Massachusetts General Hospital（麻省总医院）；Harvard Medical School（哈佛医学院）；Lehigh University（里海大学）
针对医学图像数据稀缺及现有方法难以准确呈现解剖与病理细节的问题，提出结合解剖-病理提示和视觉码本细粒度图文对齐的合成模型，并在胸部 X 射线数据上验证其语义准确性。
摘要：
数据稀缺和隐私问题限制了可供公开使用的高质量医学图像，而医学图像合成可以缓解这一问题。然而，当前的医学图像合成方法往往难以准确捕捉详细解剖结构和病理状态的复杂性。为应对这些挑战，我们提出一种新型医学图像合成模型，利用细粒度图文对齐和解剖-病理提示来生成高度细致且准确的合成医学图像。我们的方法将先进的自然语言处理技术与图像生成建模相结合，使描述性文本提示与合成图像的解剖和病理细节之间能够精确对齐。所提方法包含两个关键组件：解剖-病理提示模块和基于细粒度对齐的合成模块。解剖-病理提示模块为高质量医学图像自动生成描述性提示。为了进一步根据生成的提示合成高质量医学图像，基于细粒度对齐的合成模块为放射学数据集预定义视觉码本，并在该码本与生成的提示之间执行细粒度对齐，以获取关键图像块作为视觉线索，从而促进准确的图像合成。我们通过在公共胸部 X 射线数据集上的实验验证了方法的优越性，并证明我们的合成图像保留了准确的语义信息，使其对多种医学应用具有价值。
英文摘要：
Data scarcity and privacy concerns limit the availability of high-quality medical images for public use, which can be mitigated through medical image synthesis. However, current medical image synthesis methods often struggle to accurately capture the complexity of detailed anatomical structures and pathological conditions. To address these challenges, we propose a novel medical image synthesis model that leverages fine-grained image-text alignment and anatomy-pathology prompts to generate highly detailed and accurate synthetic medical images. Our method integrates advanced natural language processing techniques with image generative modeling, enabling precise alignment between descriptive text prompts and the synthesized images' anatomical and pathological details. The proposed approach consists of two key components: an anatomy-pathology prompting module and a fine-grained alignment-based synthesis module. The anatomy-pathology prompting module automatically generates descriptive prompts for high-quality medical images. To further synthesize high-quality medical images from the generated prompts, the fine-grained alignment-based synthesis module pre-defines a visual codebook for the radiology dataset and performs fine-grained alignment between the codebook and generated prompts to obtain key patches as visual clues, facilitating accurate image synthesis. We validate the superiority of our method through experiments on public chest X-ray datasets and demonstrate that our synthetic images preserve accurate semantic information, making them valuable for various medical applications.
Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）；Institute for AI in Medicine, University Hospital Essen（埃森大学医院人工智能医学研究所）；University of Duisburg-Essen（杜伊斯堡-埃森大学）；German Cancer Consortium (DKTK)（德国癌症联盟）
McGill University（麦吉尔大学）；MILA (Québec Artificial Intelligence Institute)（MILA（魁北克人工智能研究所））；University of Edinburgh（爱丁堡大学）；The Alan Turing Institute（艾伦·图灵研究所）
We propose a hierarchically structured variational inference model for accurately disentangling observable evidence of disease (e.g. brain lesions or atrophy) from subject-specific anatomy in brain MRIs. With flexible, partially autoregressive priors, our model (1) addresses the subtle and fine-grained dependencies that typically exist between anatomical and pathological generating factors of an MRI to ensure the clinical validity of generated samples; (2) preserves and disentangles finer pathological details pertaining to a patient's disease state. Additionally, we experiment with an alternat…
The Predictability Bias of Joint-Embedding Predictive Learning in Medical Imaging
Technical University of Munich (TUM) and TUM University Hospital（慕尼黑工业大学和慕尼黑大学医院）；Technical University of Munich, School of Medicine and Health, TUM University Hospital, Klinikum rechts der Isar, Department of Diagnostic and Interventional Radiology（慕尼黑大学医学院和健康学院，慕尼黑大学医院，伊萨尔河右岸诊所，诊断与介入放射学系）；Friedrich-Alexander University Erlangen-Nürnberg（埃尔朗根-纽伦堡弗里德里希·阿列克桑德大学）；Department of Computing, Imperial College London（伦敦帝国理工学院计算机系）；Institute for Medical AI (IMAI), Heidelberg University Hospital and Medical Faculty of Heidelberg University（海德堡大学医院医学人工智能研究所和海德堡大学医学院）；Else Kroener Fresenius Center for Digital Health (EKFZ), Faculty of Medicine Carl Gustav Carus, TUD Dresden University of Technology（德累斯顿工业大学卡尔·古斯塔夫·卡鲁斯医学院 Else Kroener Fresenius 数字健康中心）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）；Institute for Artificial Intelligence in Medicine (IKIM), University Hospital Essen (AöR)（埃森大学医院医学人工智能研究所）；Institute of Interventional and Diagnostic Radiology and Neuroradiology, University Hospital Essen（埃森大学医院介入与诊断放射学及神经放射学研究所）；National Center for Tumor Diseases West（西德国立肿瘤疾病中心）
Virginia Tech（弗吉尼亚理工大学）；IBM Research（IBM研究院）；Institute for Medical Engineering and Science, Massachusetts Institute of Technology（麻省理工学院医学工程与科学研究所）；Beth Israel Deaconess Medical Center（贝斯以色列女执事医疗中心）；Harvard T.H. Chan School of Public Health（哈佛陈曾熙公共卫生学院）；University of British Columbia（不列颠哥伦比亚大学）
Despite the progress in utilizing deep learning to automate chest radiograph interpretation and disease diagnosis tasks, change between sequential Chest X-rays (CXRs) has received limited attention. Monitoring the progression of pathologies that are visualized through chest imaging poses several challenges in anatomical motion estimation and image registration, i.e., spatially aligning the two images and modeling temporal dynamics in change detection. In this work, we propose CheXRelNet, a neural model that can track longitudinal pathology change relations between two CXRs. CheXRelNet incorpor…
MeDSLIP: Medical Dual-Stream Language-Image Pre-training with Pathology-Anatomy Semantic Alignment
Centre for Machine Intelligence and School of Computer Science, University of Sheffield（智能中心和计算机科学学院，谢菲尔德大学）；School of Medicine and Population Health, and INSIGNEO, Institute for in Silico Medicine, University of Sheffield（医学与人口健康学院，INSIGNEO，虚拟医学研究所，谢菲尔德大学）；Digital Environment Research Institute, Queen Mary University of London（数字环境研究所，伦敦大学玛丽女王学院）；School of Computer Science, University of Sheffield（计算机科学学院，谢菲尔德大学）；Department of Computing, Imperial College London（计算系，伦敦帝国学院）
A multimodal vision foundation model for generalizable knee pathology
Beihang University（北京航空航天大学）；Peking University Third Hospital（北京大学第三医院）；Institute of Sports Medicine of Peking University（北京大学运动医学研究所）；Ministry of Education（教育部）
IBM Research - Australia（IBM澳大利亚研究院）；EPF Laussane（洛桑联邦理工学院）
Anatomical landmark segmentation and pathology localization are important steps in automated analysis of medical images. They are particularly challenging when the anatomy or pathology is small, as in retinal images and cardiac MRI, or when the image is of low quality due to device acquisition parameters as in magnetic resonance (MR) scanners. We propose an image super-resolution method using progressive generative adversarial networks (P-GAN) that can take as input a low-resolution image and generate a high resolution image of desired scaling factor. The super resolved images can be used for …
Beyond Diagnosis: Evaluating Multimodal LLMs for Pathology Localization in Chest Radiographs
University of California（加州大学）；Dana-Farber Cancer Institute（达纳-法伯癌症研究所）；Massachusetts General Hospital（麻省总医院）；St. Jude Children’s Research Hospital（圣犹大儿童研究医院）；Brigham and Women’s Hospital & Harvard Medical School（布里法伦医院及哈佛医学院）
Pathological area segmentation in cardiac magnetic resonance (MR) images plays a vital role in the clinical diagnosis of cardiovascular diseases. Because of the irregular shape and small area, pathological segmentation has always been a challenging task. We propose an anatomy prior based framework, which combines the U-net segmentation network with the attention technique. Leveraging the fact that the pathology is inclusive, we propose a neighborhood penalty strategy to gauge the inclusion relationship between the myocardium and the myocardial infarction and no-reflow areas. This neighborhood …
GRASPing Anatomy to Improve Pathology Segmentation
Karlsruhe Institute of Technology, Karlsruhe, Germany（卡尔斯鲁厄理工学院）；Helmholtz Information and Data Science School for Health, Germany（海德堡信息与数据科学健康学校）；Department of Nuclear Medicine, University of Duisburg-Essen（杜伊斯堡-埃森大学核医学系）；German Cancer Consortium (DKTK)- University Hospital Essen, Essen, Germany（德国癌症研究中心（DKTK）-埃森大学医院）
Helmholtz AI（亥姆霍兹AI）；Helmholtz Center Munich（慕尼黑亥姆霍兹中心）；Klinikum Rechts der Isar（右岸医院）；Technical University of Munich (TUM)（慕尼黑工业大学）
In recent years, data-driven machine learning (ML) methods have revolutionized the computer vision community by providing novel efficient solutions to many unsolved (medical) image analysis problems. However, due to the increasing privacy concerns and data fragmentation on many different sites, existing medical data are not fully utilized, thus limiting the potential of ML. Federated learning (FL) enables multiple parties to collaboratively train a ML model without exchanging local data. However, data heterogeneity (non-IID) among the distributed clients is yet a challenge. To this end, we pro…
Radboud University Medical Center（拉德堡德大学医学中心）；Radboud Institute for Health Sciences（拉德堡德健康科学研究所）
Early detection improves prognosis in pancreatic ductal adenocarcinoma (PDAC) but is challenging as lesions are often small and poorly defined on contrast-enhanced computed tomography scans (CE-CT). Deep learning can facilitate PDAC diagnosis, however current models still fail to identify small (<2cm) lesions. In this study, state-of-the-art deep learning models were used to develop an automatic framework for PDAC detection, focusing on small lesions. Additionally, the impact of integrating surrounding anatomy was investigated. CE-CT scans from a cohort of 119 pathology-proven PDAC patients an…
Computational anatomy allows the quantitative analysis of organs in medical images. However, most analysis is constrained to the millimeter scale because of the limited resolution of clinical computed tomography (CT). X-ray microtomography ($μ$CT) on the other hand allows imaging of ex-vivo tissues at a resolution of tens of microns. In this work, we use clinical CT to image lung cancer patients before partial pneumonectomy (resection of pathological lung tissue). The resected specimen is prepared for $μ$CT imaging at a voxel resolution of 50 $μ$m (0.05 mm). This high-resolution image of the l…
University of Edinburgh（爱丁堡大学）；Edinburgh Imaging Facility QMRI（爱丁堡影像设施QMRI）；Centre for Cardiovascular Science（心血管科学中心）；Cedars Sinai Medical Center（西达赛奈医疗中心）；The Alan Turing Institute（艾伦·图灵研究所）
Typically, a medical image offers spatial information on the anatomy (and pathology) modulated by imaging specific characteristics. Many imaging modalities including Magnetic Resonance Imaging (MRI) and Computed Tomography (CT) can be interpreted in this way. We can venture further and consider that a medical image naturally factors into some spatial factors depicting anatomy and factors that denote the imaging characteristics. Here, we explicitly learn this decomposed (disentangled) representation of imaging data, focusing in particular on cardiac images. We propose Spatial Decomposition Netw…
Taiyuan University of Technology（太原理工大学）；University of Science and Technology of China（中国科学技术大学）；Second Hospital of Shanxi Medical University（山西医科大学第二医院）
Towards Segmenting the Invisible: An End-to-End Registration and Segmentation Framework for Weakly Supervised Tumour Analysis
Institute of Technical；Business Information Systems, Faculty of Computer Science, Otto von Guericke University Magdeburg, Magdeburg, Germany Human Technopole, Milan, Italy Institute of Medical Engineering, Faculty of Electrical Engineering；Information Technology, Otto von Guericke University Magdeburg, Magdeburg, Germany Centre for Behavioural Brain Sciences, Magdeburg, Germany
NVIDIA（英伟达）；DeepMind（深度思维）；Oxford Nanopore Technologies（牛津纳米孔技术公司）；University College London（伦敦大学学院）
Human anatomy, morphology, and associated diseases can be studied using medical imaging data. However, access to medical imaging data is restricted by governance and privacy concerns, data ownership, and the cost of acquisition, thus limiting our ability to understand the human body. A possible solution to this issue is the creation of a model able to learn and then generate synthetic images of the human body conditioned on specific characteristics of relevance (e.g., age, sex, and disease status). Deep generative models, in the form of neural networks, have been recently used to create synthe…
University of Strasbourg, CNRS, INSERM, ICube, UMR7357, Strasbourg, France（斯特拉斯堡大学、法国国家科学研究中心、法国国家医学研究院、ICube、UMR7357、法国斯特拉斯堡）；Department of General Surgery, Kaohsiung Chang Gung Memorial Hospital, Chang Gung University College of Medicine, Kaohsiung, Taiwan（高雄长庚纪念医院外科部、长庚大学医学院）
GALAR-TemporalNet v2: Anatomy-Guided Dual-Branch Temporal Classification with Bidirectional Mamba and Dual-Graph GCN for Video Capsule Endoscopy -- after competition results
School of Computer Science and Engineering, Kyungpook National University（韩国庆北国立大学计算机科学与工程学院）
Scientific Computing and Imaging Institute（科学计算与成像研究所）；University of Utah（犹他大学）
Statistical shape modeling (SSM) directly from 3D medical images is an underutilized tool for detecting pathology, diagnosing disease, and conducting population-level morphology analysis. Deep learning frameworks have increased the feasibility of adopting SSM in medical practice by reducing the expert-driven manual and computational overhead in traditional SSM workflows. However, translating such frameworks to clinical practice requires calibrated uncertainty measures as neural networks can produce over-confident predictions that cannot be trusted in sensitive clinical decision-making. Existin…
University of Edinburgh（爱丁堡大学）；Canon Medical Research Europe（佳能医疗欧洲研究中心）；The Alan Turing Institute（艾伦·图灵研究所）
Acquiring annotated data at scale with rare diseases or conditions remains a challenge. It would be extremely useful to have a method that controllably synthesizes images that can correct such underrepresentation. Assuming a proper latent representation, the idea of a "latent vector arithmetic" could offer the means of achieving such synthesis. A proper representation must encode the fidelity of the input data, preserve invariance and equivariance, and permit arithmetic operations. Motivated by the ability to disentangle images into spatial anatomy (tensor) factors and accompanying imaging (ve…
Agentic Large Language Models for Training-Free Neuro-Radiological Image Analysis
Department of Radiation Oncology, TUM University Hospital, Munich, Germany（放射肿瘤科，慕尼黑技术大学医院，德国）；Chair for AI in Healthcare and Medicine, Technical University of Munich (TUM)（人工智能在医疗和健康领域的主任，慕尼黑技术大学）；Chair for AI for Image-Guided Diagnosis and Therapy, Technical University of Munich (TUM)（人工智能在影像引导诊断和治疗领域的主任，慕尼黑技术大学）；Munich Center for Machine Learning (MCML), Munich, Germany（慕尼黑机器学习中心（MCML），德国慕尼黑）；Department of Computing, Imperial College London, London, UK（计算系，伦敦帝国学院，英国伦敦）；Deutsches Konsortium für Translationale Krebsforschung (DKTK), Partner Site Munich, Munich, Germany（德国转化癌症研究联盟（DKTK），慕尼黑分部，德国慕尼黑）
Guiding the Guidance: A Comparative Analysis of User Guidance Signals for Interactive Segmentation of Volumetric Images
Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）；HIDSS4Health - Helmholtz Information and Data Science School for Health（HIDSS4Health——亥姆霍兹健康信息与数据科学学院）；Institute for AI in Medicine, University Hospital Essen（埃森大学医院医学人工智能研究所）；Cancer Research Center Cologne Essen (CCCE), University Medicine Essen（埃森大学医学院科隆埃森癌症研究中心（CCCE））
RadImageNet-VQA: A Large-Scale CT and MRI Dataset for Radiologic Visual Question Answering
Raidium；Université de Paris Cité, Hôpital Européen Georges Pompidou, AP-HP（巴黎西岱大学，乔治·蓬皮杜欧洲医院，AP-HP）；Department of Vascular and Oncological Interventional Radiology, INSERM（血管与肿瘤介入放射学系，法国国家健康与医学研究院）
Mask-guided Data Augmentation for Multiparametric MRI Generation with a Rare Hepatocellular Carcinoma
Universidad Industrial de Santander（桑坦德工业大学）；University of Toulouse（图卢兹大学）；Université Paul Sabatier Toulouse 3（图卢兹第三大学（保罗·萨巴蒂埃大学））；Bicêtre University Hospital（比塞特尔大学医院）；Assistance publique des hôpitaux de Paris（巴黎公共援助医院）；Université Claude Bernard Lyon 1（里昂第一大学（克洛德·贝尔纳里昂第一大学））
Wasserstein-Aligned Localisation for VLM-Based Distributional OOD Detection in Medical Imaging
Department of Computing, Imperial College London, UK（伦敦帝国理工学院计算机系）；Technical University Munich, DE（慕尼黑技术大学）；Munich Center for Machine Learning (MCML), DE（慕尼黑机器学习中心（MCML））
Specialty-Oriented Generalist Medical AI for Chest CT Screening
Rensselaer Polytechnic Institute（伦斯勒理工学院）；Wake Forest University School of Medicine（维克森林大学医学院）；Massachusetts General Hospital（麻省总医院）；Harvard Medical School（哈佛医学院）
MedSyn2: Flexible Control of 3D CT Generation via Text and Semantically-Defined Segmentation Prompts
Boston University School of Engineering（波士顿大学工程学院）；Stanford University（斯坦福大学）；University of Pittsburgh Medical Center（匹兹堡大学医学中心）；Boston University School of Medicine（波士顿大学医学院）
