F3-Net is a foundation model designed to overcome persistent challenges in clinical medical image segmentation, including reliance on complete multimodal inputs, limited generalizability, and narrow task specificity. Through flexible synthetic modality training, F3-Net maintains robust performance even in the presence of missing MRI sequences, leveraging a zero-image strategy to substitute absent modalities without relying on explicit synthesis networks, thereby enhancing real-world applicability. Its unified architecture supports multi-pathology segmentation across glioma, metastasis, stroke, and white matter lesions without retraining, outperforming CNN-based and transformer-based models that typically require disease-specific fine-tuning. Evaluated on diverse datasets such as BraTS 2021, BraTS 2024, and ISLES 2022, F3-Net demonstrates strong resilience to domain shifts and clinical heterogeneity. On the whole pathology dataset, F3-Net achieves average Dice Similarity Coefficients (DSCs) of 0.94 for BraTS-GLI 2024, 0.82 for BraTS-MET 2024, 0.94 for BraTS 2021, and 0.79 for ISLES 2022. This positions it as a versatile, scalable solution bridging the gap between deep learning research and practical clinical deployment.
Isfahan University of Technology（伊斯法罕理工大学）；University of Michigan（密歇根大学）
Automating classification and segmentation process of abnormal regions in different body organs has a crucial role in most of medical imaging applications such as funduscopy, endoscopy, and dermoscopy. Detecting multiple abnormalities in each type of images is necessary for better and more accurate diagnosis procedure and medical decisions. In recent years portable medical imaging devices such as capsule endoscopy and digital dermatoscope have been introduced and made the diagnosis procedure easier and more efficient. However, these portable devices have constrained power resources and limited…
X2CT-CLIP: Enable Multi-Abnormality Detection in Computed Tomography from Chest Radiography via Tri-Modal Contrastive Learning
Peter Munk Cardiac Centre（彼得·芒克心脏中心）；University Health Network (UHN)（大学健康网络）；University of Toronto (U of T)（多伦多大学）；Ted Rogers Centre for Heart Research（泰德·罗杰斯心脏研究中心）；Toronto General Hospital Research Institute（多伦多综合医院研究所）；Vector Institute（矢量研究所）
Exploring Self-Supervised Learning with U-Net Masked Autoencoders and EfficientNet-B7 for Improved Gastrointestinal Abnormality Classification in Video Capsule Endoscopy
International Institute of Information Technology, Bangalore（国际信息技术学院，班加罗尔）；Indian Institute of Technology, Hyderabad（印度理工学院，海得拉巴）
Chest X-Rays (CXRs) are widely used for diagnosing abnormalities in the heart and lung area. Automatically detecting these abnormalities with high accuracy could greatly enhance real world diagnosis processes. Lack of standard publicly available dataset and benchmark studies, however, makes it difficult to compare various detection methods. In order to overcome these difficulties, we have used a publicly available Indiana CXR, JSRT and Shenzhen dataset and studied the performance of known deep convolutional network (DCN) architectures on different abnormalities. We find that the same DCN archi…
The University of Texas at Austin（德克萨斯大学奥斯汀分校）；Icahn School of Medicine at Mount Sinai（西奈山伊坎医学院）；Weill Cornell Medicine（威尔康奈尔医学院）
Building a highly accurate predictive model for classification and localization of abnormalities in chest X-rays usually requires a large number of manually annotated labels and pixel regions (bounding boxes) of abnormalities. However, it is expensive to acquire such annotations, especially the bounding boxes. Recently, contrastive learning has shown strong promise in leveraging unlabeled natural images to produce highly generalizable and discriminative features. However, extending its power to the medical image domain is under-explored and highly non-trivial, since medical images are much les…
University of Pittsburgh（匹兹堡大学）；University of Pittsburgh Medical Center（匹兹堡大学医学中心）
Creating a large-scale dataset of abnormality annotation on medical images is a labor-intensive and costly task. Leveraging weak supervision from readily available data such as radiology reports can compensate lack of large-scale data for anomaly detection methods. However, most of the current methods only use image-level pathological observations, failing to utilize the relevant anatomy mentions in reports. Furthermore, Natural Language Processing (NLP)-mined weak labels are noisy due to label sparsity and linguistic ambiguity. We propose an Anatomy-Guided chest X-ray Network (AGXNet) to addr…
Virginia Polytechnic Institute and State University (Virginia Tech)（弗吉尼亚理工大学（Virginia Tech））；Department of Mechanical Engineering（机械工程系）；Autonomous Systems and Intelligent Machines (ASIM) Lab（自主系统与智能机器（ASIM）实验室）；Graduate Research Assistant（研究生研究助理）
This paper introduces MuRAD (Musculoskeletal Radiograph Abnormality Detection tool), a tool that can help radiologists automate the detection of abnormalities in musculoskeletal radiographs (bone X-rays). MuRAD utilizes a Convolutional Neural Network (CNN) that can accurately predict whether a bone X-ray is abnormal, and leverages Class Activation Map (CAM) to localize the abnormality in the image. MuRAD achieves an F1 score of 0.822 and a Cohen's kappa of 0.699, which is comparable to the performance of expert radiologists.
Institute for Infocomm Research, A*STAR, Singapore（信息通信研究所（A*STAR，新加坡））
Supervised deep learning algorithms have enabled significant performance gains in medical image classification tasks. But these methods rely on large labeled datasets that require resource-intensive expert annotation. Semi-supervised generative adversarial network (GAN) approaches offer a means to learn from limited labeled data alongside larger unlabeled datasets, but have not been applied to discern fine-scale, sparse or localized features that define medical abnormalities. To overcome these limitations, we propose a patch-based semi-supervised learning approach and evaluate performance on c…
3D Modality-Aware Pre-training for Vision-Language Model in MRI Multi-organ Abnormality Detection
School of Electronic, Electrical Engineering and Physics, Fujian University of Technology（福建工程学院电子电气工程学院）；School of Computer Science and Engineering, Southeast University, China（东南大学计算机科学与工程学院）；Department of Medical Imaging, Suzhou Traditional Chinese Medicine Hospital, China（苏州中医医院影像科）
Hong Kong Baptist University（香港浸会大学）；Queen Elizabeth and Kwong Wah Hospitals（伊利沙伯医院及广华医院）；Tuen Mun Hospital（屯门医院）；NVIDIA Corporation（英伟达公司）
Automatic generation of medical reports from X-ray images can assist radiologists to perform the time-consuming and yet important reporting task. Yet, achieving clinically accurate generated reports remains challenging. Modeling the underlying abnormalities using the knowledge graph approach has been found promising in enhancing the clinical accuracy. In this paper, we introduce a novel fined-grained knowledge graph structure called an attributed abnormality graph (ATAG). The ATAG consists of interconnected abnormality nodes and attribute nodes, allowing it to better capture the abnormality de…
Learning Global and Local Features of Normal Brain Anatomy for Unsupervised Abnormality Detection
National Cancer Center Research Institute（国家癌症中心研究所）；RIKEN Center for Advanced Intelligence Project（理化学研究所先进智能项目中心）；Graduate School of Information Science and Technology, The University of Tokyo（东京大学信息科学与技术研究生院）；Research Center for Advanced Science and Technology, The University of Tokyo（东京大学先进科学技术研究中心）；National Cancer Center Hospital（国家癌症中心医院）
Understanding model predictions is critical in healthcare, to facilitate rapid verification of model correctness and to guard against use of models that exploit confounding variables. We introduce the challenging new task of explainable multiple abnormality classification in volumetric medical images, in which a model must indicate the regions used to predict each abnormality. To solve this task, we propose a multiple instance learning convolutional neural network, AxialNet, that allows identification of top slices for each abnormality. Next we incorporate HiResCAM, an attention mechanism, to …
Athinoula A. Martinos Center for Biomedical Imaging（阿西诺拉·A·马蒂诺斯生物医学成像中心）；Massachusetts General Hospital（麻省总医院）；Arizona State University-Mayo Clinic Center for Innovative Imaging（亚利桑那州立大学-梅奥诊所创新成像中心）；School of Computing, Informatics, and Decision Systems Engineering（计算、信息学与决策系统工程学院）
Saliency maps have become a widely used method to assess which areas of the input image are most pertinent to the prediction of a trained neural network. However, in the context of medical imaging, there is no study to our knowledge that has examined the efficacy of these techniques and quantified them using overlap with ground truth bounding boxes. In this work, we explored the credibility of the various existing saliency map methods on the RSNA Pneumonia dataset. We found that GradCAM was the most sensitive to model parameter and label randomization, and was highly agnostic to model architec…
Suppression of thoracic bone shadows on chest X-rays (CXRs) has been indicated to improve the diagnosis of pulmonary disease. Previous approaches can be categorized as unsupervised physical and supervised deep learning models. Nevertheless, with physical models able to preserve morphological details but at the cost of extremely long processing time, existing DL methods face challenges of gathering sufficient/qualitative ground truth (GT) for robust training, thus leading to failure in maintaining clinically acceptable false positive rates. We hereby propose a generalizable yet efficient workfl…
Google-MedGemma Based Abnormality Detection in Musculoskeletal radiographs
University of Texas at Arlington, Texas, United States（德克萨斯大学阿灵顿分校）；University of Engineering and Management, Kolkata, India（工程与管理大学）；Sardar Vallabhbhai National Institute of Technology, Surat, India（萨达尔·瓦拉布尔希国家理工学院）
Mekelle Institute of Technology - Mekelle University（梅克莱大学梅克莱理工学院）；FDRE Ministry of Innovation and Technology（埃塞俄比亚联邦民主共和国创新与技术部）；College of Health Science - Mekelle University（梅克莱大学健康科学学院）
Breast cancer is prevalent in Ethiopia that accounts 34% among women cancer patients. The diagnosis technique in Ethiopia is manual which was proven to be tedious, subjective, and challenging. Deep learning techniques are revolutionizing the field of medical image analysis and hence in this study, we proposed Convolutional Neural Networks (CNNs) for breast mass detection so as to minimize the overheads of manual analysis. CNN architecture is designed for the feature extraction stage and adapted both the Region Proposal Network (RPN) and Region of Interest (ROI) portion of the faster R-CNN for …
Machine learning models for radiology benefit from large-scale data sets with high quality labels for abnormalities. We curated and analyzed a chest computed tomography (CT) data set of 36,316 volumes from 19,993 unique patients. This is the largest multiply-annotated volumetric medical imaging data set reported. To annotate this data set, we developed a rule-based method for automatically extracting abnormality labels from free-text radiology reports with an average F-score of 0.976 (min 0.941, max 1.0). We also developed a model for multi-organ, multi-disease classification of chest CT volum…
CXRs are a crucial and extraordinarily common diagnostic tool, leading to heavy research for CAD solutions. However, both high classification accuracy and meaningful model predictions that respect and incorporate clinical taxonomies are crucial for CAD usability. To this end, we present a deep HMLC approach for CXR CAD. Different than other hierarchical systems, we show that first training the network to model conditional probability directly and then refining it with unconditional probabilities is key in boosting performance. In addition, we also formulate a numerically stable cross-entropy l…
Shanghai Jiao Tong University（上海交通大学）；Shandong Cancer Hospital Affiliated to Shandong University（山东大学附属山东省肿瘤医院）；Deepwise AI Lab（深睿医疗人工智能实验室）
Segmentation of nasopharyngeal carcinoma (NPC) from Magnetic Resonance Images (MRI) is a crucial prerequisite for NPC radiotherapy. However, manually segmenting of NPC is time-consuming and labor-intensive. Additionally, single-modality MRI generally cannot provide enough information for its accurate delineation. Therefore, a multi-modality MRI fusion network (MMFNet) based on three modalities of MRI (T1, T2 and contrast-enhanced T1) is proposed to complete accurate segmentation of NPC. The backbone of MMFNet is designed as a multi-encoder-based network, consisting of several encoders to captu…
College of Physics and Information Engineering, Fuzhou University（福州大学物理与信息工程学院）；School of Data Science, Fudan University（复旦大学大数据学院）；School of Biomedical Engineering, Shanghai Jiao Tong University（上海交通大学生物医学工程学院）
Multi-modality medical images can provide relevant or complementary information for a target (organ, tumor or tissue). Registering multi-modality images to a common space can fuse these comprehensive information, and bring convenience for clinical application. Recently, neural networks have been widely investigated to boost registration methods. However, it is still challenging to develop a multi-modality registration network due to the lack of robust criteria for network training. In this work, we propose a multi-modality registration network (MMRegNet), which can perform registration between…
Cross-modality image estimation involves the generation of images of one medical imaging modality from that of another modality. Convolutional neural networks (CNNs) have been shown to be useful in identifying, characterising and extracting image patterns. Generative adversarial networks (GANs) use CNNs as generators and estimated images are discriminated as true or false based on an additional network. CNNs and GANs within the image estimation framework may be considered more generally as deep learning approaches, since imaging data tends to be large, leading to a larger number of network wei…
DM-QPMNET: Dual-modality fusion network for cell segmentation in quantitative phase microscopy
College of Computing and Informatics, University of North Carolina at Charlotte, NC, USA（计算与信息学院，北卡罗来纳大学夏洛特分校）；Department of Physics and Optical Science, University of North Carolina at Charlotte, NC, USA（物理与光学科学系，北卡罗来纳大学夏洛特分校）；Center for TAIMing AI, University of North Carolina at Charlotte, NC, USA（TAIMing AI中心，北卡罗来纳大学夏洛特分校）
Memorial Sloan Kettering Cancer Center（纪念斯隆·凯特琳癌症中心）
Lung tumors, especially those located close to or surrounded by soft tissues like the mediastinum, are difficult to segment due to the low soft tissue contrast on computed tomography images. Magnetic resonance images contain superior soft-tissue contrast information that can be leveraged if both modalities were available for training. Therefore, we developed a cross-modality educed learning approach where MR information that is educed from CT is used to hallucinate MRI and improve CT segmentation. Our approach, called cross-modality educed deep learning segmentation (CMEDL) combines CT and pse…
Carleton University（卡尔顿大学）；National Research Council Canada（加拿大国家研究委员会）
Breast cancer is the most common cancer in women worldwide. The most common screening technology is mammography. To reduce the cost and workload of radiologists, we propose a computer aided detection approach for classifying and localizing calcifications and masses in mammogram images. To improve on conventional approaches, we apply deep convolutional neural networks (CNN) for automatic feature learning and classifier building. In computer-aided mammography, deep CNN classifiers cannot be trained directly on full mammogram images because of the loss of image details from resizing at input laye…
In this paper, we describe our method for classification of brain magnetic resonance (MR) images into different abnormalities and healthy classes based on the deep neural network. We propose our method to detect high and low-grade glioma, multiple sclerosis, and Alzheimer diseases as well as healthy cases. Our network architecture has ten learning layers that include seven convolutional layers and three fully connected layers. We have achieved a promising result in five categories of brain images (classification task) with 95.7% accuracy.
PDPM Indian Institute of Information Technology, Design and Manufacturing（PDPM印度信息技术设计与制造学院）
Wireless Capsule Endoscopy is one of the most advanced non-invasive methods for the examination of gastrointestinal tracts. An intelligent computer-aided diagnostic system for detecting gastrointestinal abnormalities like polyp, bleeding, inflammation, etc. is highly exigent in wireless capsule endoscopy image analysis. Abnormalities greatly differ in their shape, size, color, and texture, and some appear to be visually similar to normal regions. This poses a challenge in designing a binary classifier due to intra-class variations. In this study, a hybrid convolutional neural network is propos…
We present a novel Convolutional Neural Network (CNN) based approach for one class classification. The idea is to use a zero centered Gaussian noise in the latent space as the pseudo-negative class and train the network using the cross-entropy loss to learn a good representation as well as the decision boundary for the given class. A key feature of the proposed approach is that any pre-trained CNN can be used as the base network for one class classification. The proposed One Class CNN (OC-CNN) is evaluated on the UMDAA-02 Face, Abnormality-1001, FounderType-200 datasets. These datasets are rel…
Segmentation has been a major task in neuroimaging. A large number of automated methods have been developed for segmenting healthy and diseased brain tissues. In recent years, deep learning techniques have attracted a lot of attention as a result of their high accuracy in different segmentation problems. We present a new deep learning based segmentation method, DeepMRSeg, that can be applied in a generic way to a variety of segmentation tasks. The proposed architecture combines recent advances in the field of biomedical image segmentation and computer vision. We use a modified UNet architectur…
Purdue University（普渡大学）；Ahsanullah University of Science and Technology（阿萨努拉科技大学）
Anomaly detection (AD) in a surveillance scenario is an emerging and challenging field of research. For autonomous vehicles like drones or cars, it is immensely important to distinguish between normal and abnormal states in real-time. Additionally, we also need to detect any device malfunction. But the nature and degree of abnormality may vary depending upon the actual environment and adversary. As a result, it is impractical to model all cases a-priori and use supervised methods to classify. Also, an autonomous vehicle provides various data types like images and other analog or digital sensor…
In this work, we present an end-to-end deep learning framework for X-ray image diagnosis. As the first step, our system determines whether a submitted image is an X-ray or not. After it classifies the type of the X-ray, it runs the dedicated abnormality classification network. In this work, we only focus on the chest X-rays for abnormality classification. However, the system can be extended to other X-ray types easily. Our deep learning classifiers are based on DenseNet-121 architecture. The test set accuracy obtained for 'X-ray or Not', 'X-ray Type Classification', and 'Chest Abnormality Clas…
University of Bristol（布里斯托大学）；Chulalongkorn University（朱拉隆功大学）
Identification of abnormalities in red blood cells (RBC) is key to diagnosing a range of medical conditions from anaemia to liver disease. Currently this is done manually, a time-consuming and subjective process. This paper presents an automated process utilising the advantages of machine learning to increase capacity and standardisation of cell abnormality detection, and its performance is analysed. Three different machine learning technologies were used: a Support Vector Machine (SVM), a classical machine learning technology; TabNet, a deep learning architecture for tabular data; U-Net, a se…
Multi-modality image registration is one of the most underlined processes in medical image analysis. Recently, convolutional neural networks (CNNs) have shown significant potential in deformable registration. However, the lack of voxel-wise ground truth challenges the training of CNNs for an accurate registration. In this work, we propose a cross-modality similarity metric, based on the KL-divergence of image variables, and implement an efficient estimation method using a CNN. This estimation network, referred to as KLDivNet, can be trained unsupervisedly. We then embed the KLDivNet into a reg…
Vanderbilt University（范德堡大学）；Vanderbilt University Medical Center（范德堡大学医学中心）；Incyte Corporation（Incyte公司）
A key limitation of deep convolutional neural networks (DCNN) based image segmentation methods is the lack of generalizability. Manually traced training images are typically required when segmenting organs in a new imaging modality or from distinct disease cohort. The manual efforts can be alleviated if the manually traced images in one imaging modality (e.g., MRI) are able to train a segmentation network for another imaging modality (e.g., CT). In this paper, we propose an end-to-end synthetic segmentation network (SynSeg-Net) to train a segmentation network for a target imaging modality with…
Combining the respective advantages of cross-modality images can compensate for the lack of information in the single modality, which has attracted increasing attention of researchers into multi-modal image matching tasks. Meanwhile, due to the great appearance differences between cross-modality image pairs, it often fails to make the feature representations of correspondences as close as possible. In this letter, we design a cross-modality feature representation learning network, S2-Net, which is based on the recently successful detect-and-describe pipeline, originally proposed for visible im…
DMAF-Net: An Effective Modality Rebalancing Framework for Incomplete Multi-Modal Medical Image Segmentation
College of Computer Science and Engineering, Chongqing University of Technology（重庆理工大学计算机科学与工程学院）；School of Computer Science and Engineering, Southeast University（东南大学计算机科学与工程学院）
For the visible-infrared person re-identification (VIReID) task, one of the major challenges is the modality gaps between visible (VIS) and infrared (IR) images. However, the training samples are usually limited, while the modality gaps are too large, which leads that the existing methods cannot effectively mine diverse cross-modality clues. To handle this limitation, we propose a novel augmentation network in the embedding space, called diverse embedding expansion network (DEEN). The proposed DEEN can effectively generate diverse embeddings to learn the informative feature representations and…
Northwestern Polytechnical University（西北工业大学）；University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）；Shanghai United Imaging Intelligence Co., Ltd（上海联影智能有限公司）；Shanghai Jiao Tong University（上海交通大学）
Non-rigid inter-modality registration can facilitate accurate information fusion from different modalities, but it is challenging due to the very different image appearances across modalities. In this paper, we propose to train a non-rigid inter-modality image registration network, which can directly predict the transformation field from the input multimodal images, such as CT and MR images. In particular, the training of our inter-modality registration network is supervised by intra-modality similarity metric based on the available paired data, which is derived from a pre-aligned CT and MR da…
Beijing Jiaotong University（北京交通大学）；Nanyang Technological University（南洋理工大学）；Sun Yat-sen University（中山大学）；University of Chinese Academy of Sciences（中国科学院大学）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；Peng Cheng Laboratory（鹏城实验室）
Focusing on the issue of how to effectively capture and utilize cross-modality information in RGB-D salient object detection (SOD) task, we present a convolutional neural network (CNN) model, named CIR-Net, based on the novel cross-modality interaction and refinement. For the cross-modality interaction, 1) a progressive attention guided integration unit is proposed to sufficiently integrate RGB-D feature representations in the encoder stage, and 2) a convergence aggregation structure is proposed, which flows the RGB and depth decoding features into the corresponding RGB-D decoding streams via …
A Unified Attention U-Net Framework for Cross-Modality Tumor Segmentation in MRI and CT
Department of Computer Science, New Mexico State University（计算机科学系，新墨西哥州立大学）；Department of Plant and Environmental Sciences, New Mexico State University（植物与环境科学系，新墨西哥州立大学）
Fuzhou University（福州大学）；Fudan University（复旦大学）；Shanghai Jiao Tong University（上海交通大学）；King’s College London（伦敦国王学院）
Both image registration and label fusion in the multi-atlas segmentation (MAS) rely on the intensity similarity between target and atlas images. However, such similarity can be problematic when target and atlas images are acquired using different imaging protocols. High-level structure information can provide reliable similarity measurement for cross-modality images when cooperating with deep neural networks (DNNs). This work presents a new MAS framework for cross-modality images, where both image registration and label fusion are achieved by DNNs. For image registration, we propose a consiste…
Normandie Univ（诺曼底大学）；UNICAEN（卡昂大学）；ENSICAEN（卡昂国立高等工程师学院）；CNRS -- UMR GREYC（法国国家科学研究中心——GREYC联合研究单位）
Multimodal patch matching addresses the problem of finding the correspondences between image patches from two different modalities, e.g. RGB vs sketch or RGB vs near-infrared. The comparison of patches of different modalities can be done by discovering the information common to both modalities (Siamese like approaches) or the modality-specific information (Pseudo-Siamese like approaches). We observed that none of these two scenarios is optimal. This motivates us to propose a three-stream architecture, dubbed as TS-Net, combining the benefits of the two. In addition, we show that adding extra c…
Recent learning-based image fusion methods have marked numerous progress in pre-registered multi-modality data, but suffered serious ghosts dealing with misaligned multi-modality data, due to the spatial deformation and the difficulty narrowing cross-modality discrepancy. To overcome the obstacles, in this paper, we present a robust cross-modality generation-registration paradigm for unsupervised misaligned infrared and visible image fusion (IVIF). Specifically, we propose a Cross-modality Perceptual Style Transfer Network (CPSTN) to generate a pseudo infrared image taking a visible image as i…
Institute of Imaging and Computer Vision, RWTH Aachen University（亚琛工业大学成像与计算机视觉研究所）
Gliomas are one of the most prevalent types of primary brain tumours, accounting for more than 30\% of all cases and they develop from the glial stem or progenitor cells. In theory, the majority of brain tumours could well be identified exclusively by the use of Magnetic Resonance Imaging (MRI). Each MRI modality delivers distinct information on the soft tissue of the human brain and integrating all of them would provide comprehensive data for the accurate segmentation of the glioma, which is crucial for the patient's prognosis, diagnosis, and determining the best follow-up treatment. Unfortun…
Explicit Abnormality Extraction for Unsupervised Motion Artifact Reduction in Magnetic Resonance Imaging
School of Electrical Engineering and Automation, Wuhan University（武汉大学电气与自动化学院）；University Hospital Heidelberg（海德堡大学医院）；Vitalent Consulting（维塔伦特咨询公司）；College of Science and Engineering, James Cook University（詹姆斯库克大学科学与工程学院）；Uppsala University（乌普萨拉大学）；School of Computer Science, University of Sydney（悉尼大学计算机学院）
