StructDiff: Structure-aware Diffusion Model for 3D Fine-grained Medical Image Synthesis
Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China（新一代人工智能技术及跨学科应用关键实验室（东南大学），中华人民共和国教育部，中国）；Jiangsu Province Joint International Research Laboratory of Medical Information Processing, Southeast University, Nanjing, China（江苏省医疗信息处理联合国际研究实验室（东南大学），南京市，中国）；Univ Rennes, CHU Rennes, Inserm, LTSI– UMR 1099, F-35000 Rennes, France（里昂大学，里昂大学医院，Inserm，LTSI– UMR 1099，法国里昂，法国）；Shanghai AI Laboratory（上海人工智能实验室）；School of Computer Science and Technology, Beijing Institute of Technology, Beijing, China（计算机科学与技术学院，北京理工大学，北京，中国）；Department of Radiology, Jinling Hospital, Affiliated Hospital of Medical School, Nanjing University, Nanjing, China（放射科，南京大学附属医院，南京大学，南京，中国）
Solving medical imaging data scarcity through semantic image generation has attracted growing attention in recent years. However, existing generative models mainly focus on synthesizing whole-organ or large-tissue structures, showing limited capability in reproducing fine-grained anatomical details. Due to the stringent requirement of topological consistency and the complex 3D morphological heterogeneity of medical data, accurately reconstructing fine-grained anatomical details remains a significant challenge. To address these limitations, we propose StructDiff, a Structure-aware Diffusion Model for fine-grained 3D medical image synthesis, which enables precise generation of topologically complex anatomies. In addition to the conventional mask-based guidance, StructDiff further introduces a paired image-mask template to guide the generation process, providing structural constrains and offering explicit knowledge of mask-to-image correspondence. Moreover, a Mask Generation Module (MGM) is designed to enrich mask diversity and alleviate the scarcity of high-quality reference masks. Furthermore, we propose a Confidence-aware Adaptive Learning (CAL) strategy based on Skip-Sampling Variance (SSV), which mitigates uncertainty introduced by imperfect synthetic data when transferring to downstream tasks. Extensive experiments demonstrate that StructDiff achieves state-of-the-art performance in terms of topological consistency and visual realism, and significantly boosts downstream segmentation performance. Code will be released upon acceptance.
Visually-Guided Controllable Medical Image Generation via Fine-Grained Semantic Disentanglement
Computer Science and Engineering, Northeastern University, Shenyang, China（东北大学计算机科学与工程系，中国沈阳）；Key Laboratory of Intelligent Computing in Medical Image of Ministry of Education, Northeastern University, Shenyang, China（教育部医学图像智能计算重点实验室，东北大学，中国沈阳）；National Frontiers Science Center for Industrial Intelligence and Systems Optimization, Shenyang, China（工业智能与系统优化国家级前沿科学中心，中国沈阳）；AiShiWeiLai AI Research, China（艾世维来人工智能研究，中国）；Amii, University of Alberta, Edmonton, Alberta, Canada（阿尔伯塔大学艾米人工智能研究所，加拿大埃德蒙顿，阿尔伯塔）
Massachusetts General Hospital（麻省总医院）；Casey Eye Institute（凯西眼科研究所）；Brigham and Women’s Hospital（布莱根妇女医院）
Generative adversarial networks (GANs) are a class of unsupervised machine learning algorithms that can produce realistic images from randomly-sampled vectors in a multi-dimensional space. Until recently, it was not possible to generate realistic high-resolution images using GANs, which has limited their applicability to medical images that contain biomarkers only detectable at native resolution. Progressive growing of GANs is an approach wherein an image generator is trained to initially synthesize low resolution synthetic images (8x8 pixels), which are then fed to a discriminator that distin…
University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）；Shanghai Jiao Tong University（上海交通大学）
Medical imaging plays a critical role in various clinical applications. However, due to multiple considerations such as cost and risk, the acquisition of certain image modalities could be limited. To address this issue, many cross-modality medical image synthesis methods have been proposed. However, the current methods cannot well model the hard-to-synthesis regions (e.g., tumor or lesion regions). To address this issue, we propose a simple but effective strategy, that is, we propose a dual-discriminator (dual-D) adversarial learning system, in which, a global-D is used to make an overall eval…
Recently, medical report generation, which aims to automatically generate a long and coherent descriptive paragraph of a given medical image, has received growing research interests. Different from the general image captioning tasks, medical report generation is more challenging for data-driven neural models. This is mainly due to 1) the serious data bias: the normal visual regions dominate the dataset over the abnormal visual regions, and 2) the very long sequence. To alleviate above two problems, we propose an AlignTransformer framework, which includes the Align Hierarchical Attention (AHA) …
Unsupervised Patch-GAN with Targeted Patch Ranking for Fine-Grained Novelty Detection in Medical Imaging
the Southern University of Science and Technology（南方科技大学）；University of Oxford（牛津大学）；Tsinghua University（清华大学）；University of Warwick（华威大学）；Northwest University（西北大学）；Peng Cheng Lab（鹏城实验室）
Arizona State University（亚利桑那州立大学）；Stanford University（斯坦福大学）；Mayo Clinic（梅奥诊所）
Transfer learning from supervised ImageNet models has been frequently used in medical image analysis. Yet, no large-scale evaluation has been conducted to benchmark the efficacy of newly-developed pre-training techniques for medical image analysis, leaving several important questions unanswered. As the first step in this direction, we conduct a systematic study on the transferability of models pre-trained on iNat2021, the most recent large-scale fine-grained dataset, and 14 top self-supervised ImageNet models on 7 diverse medical tasks in comparison with the supervised ImageNet model. Furtherm…
Arizona State University（亚利桑那州立大学）；Mayo Clinic（梅奥诊所）
Discriminative learning, restorative learning, and adversarial learning have proven beneficial for self-supervised learning schemes in computer vision and medical imaging. Existing efforts, however, omit their synergistic effects on each other in a ternary setup, which, we envision, can significantly benefit deep semantic representation learning. To realize this vision, we have developed DiRA, the first framework that unites discriminative, restorative, and adversarial learning in a unified manner to collaboratively glean complementary visual information from unlabeled medical images for fine-…
We propose a novel transformer model, capable of segmenting medical images of varying modalities. Challenges posed by the fine grained nature of medical image analysis mean that the adaptation of the transformer for their analysis is still at nascent stages. The overwhelming success of the UNet lay in its ability to appreciate the fine-grained nature of the segmentation task, an ability which existing transformer based models do not currently posses. To address this shortcoming, we propose The Fully Convolutional Transformer (FCT), which builds on the proven ability of Convolutional Neural Net…
This paper considers making active learning more sensible from a medical perspective. In practice, a disease manifests itself in different forms across patient cohorts. Existing frameworks have primarily used mathematical constructs to engineer uncertainty or diversity-based methods for selecting the most informative samples. However, such algorithms do not present themselves naturally as usable by the medical community and healthcare providers. Thus, their deployment in clinical settings is very limited, if any. For this purpose, we propose a framework that incorporates clinical insights into…
Semantic Segmentation for Preoperative Planning in Transcatheter Aortic Valve Replacement
Karlsruhe Institute of Technology, Karlsruhe, Germany（卡尔斯鲁厄理工学院）；Helmholtz Information and Data Science School for Health, Germany（海德堡信息与数据科学健康学校）；Department of Thoracic and Cardiovascular Surgery, Heart Centre Lahr, Germany（胸外科与心血管外科部门，劳赫心脏中心）
As an essential prerequisite for developing a medical intelligent assistant system, medical image segmentation has received extensive research and concentration from the neural network community. A series of UNet-like networks with encoder-decoder architecture has achieved extraordinary success, in which UNet2+ and UNet3+ redesign skip connections, respectively proposing dense skip connection and full-scale skip connection and dramatically improving compared with UNet in medical image segmentation. However, UNet2+ lacks sufficient information explored from the full scale, which will affect the…
Application of attention-based Siamese composite neural network in medical image recognition
College of Computer Science and Electronic Engineering（计算机与电子工程学院）；Hunan University（湖南大学）；College of Material Science and Engineering（材料科学与工程学院）；School of Basic Education（基础教育学院）；Changsha Aeronautical Vocational and Technical College（长沙航空职业技术学院）
Phi-SegNet: Phase-Integrated Supervision for Medical Image Segmentation
mHealth Lab, Department of Biomedical Engineering, Bangladesh University of Engineering and Technology（孟加拉工程与技术大学生物医学工程系mHealth实验室）；Department of Electrical and Computer Engineering, Johns Hopkins University（约翰霍普金斯大学电气与计算机工程系）；Center for Bioengineering Innovation and Design, Department of Biomedical Engineering, Johns Hopkins University（约翰霍普金斯大学生物工程创新与设计中心，生物医学工程系）
Mitsubishi Electric Research Laboratories (MERL)（三菱电机研究实验室（MERL））
Pothole classification has become an important task for road inspection vehicles to save drivers from potential car accidents and repair bills. Given the limited computational power and fixed number of training epochs, we propose iterative self knowledge distillation (ISKD) to train lightweight pothole classifiers. Designed to improve both the teacher and student models over time in knowledge distillation, ISKD outperforms the state-of-the-art self knowledge distillation method on three pothole classification datasets across four lightweight network architectures, which supports that self know…
An Empirical Analysis for Zero-Shot Multi-Label Classification on COVID-19 CT Scans and Uncurated Reports
ARTORG Center for Biomedical Engineering Research（ARTORG生物医学工程研究中心）；University of Bern（伯尔尼大学）；Bern University Hospital（伯尔尼大学医院）；Lindenhofspital（林登霍夫医院）；Campus Stiftung Lindenhof Bern（伯尔尼林登霍夫基金会校区）
Improving Representation of High-frequency Components for Medical Visual Foundation Models
King Abdullah University of Science and Technology (KAUST)（阿卜杜拉国王科技大学）；Ningbo Institute of Information Technology Application, Chinese Academy of Sciences (CAS)（中国科学院宁波信息技术应用研究所）
Beyond the common difficulties faced in the natural image captioning, medical report generation specifically requires the model to describe a medical image with a fine-grained and semantic-coherence paragraph that should satisfy both medical commonsense and logic. Previous works generally extract the global image features and attempt to generate a paragraph that is similar to referenced reports; however, this approach has two limitations. Firstly, the regions of primary interest to radiologists are usually located in a small area of the global image, meaning that the remainder parts of the ima…
