本文提出 Malenia 框架，通过多尺度掩码-属性对齐和跨模态知识注入模块，将已见病灶的可扩展知识与未见病灶视觉特征链接，有效解决 3D CT 零样本病灶分割任务。
摘要：
医学视觉-语言预训练模型的最新进展推动了零样本疾病识别的显著进步。然而，将图像级知识迁移至像素级任务（如 3D CT 扫描中的病灶分割）仍是一项关键挑战。由于病理视觉特征的复杂性与多变性，现有方法难以将训练中未遇到的细粒度病灶特征与疾病相关的文本表示进行对齐。本文提出 Malenia，一种新颖的多尺度病灶级掩码-属性对齐框架，专为 3D 零样本病灶分割设计。Malenia 提升了掩码表示与其关联元素属性间的兼容性，明确将未见病灶的视觉特征与从已见病灶学到的可扩展知识相链接。此外，我们设计了 Cross-Modal Knowledge Injection 模块，利用互益信息增强视觉与文本特征，有效引导分割结果的生成。跨三个数据集和 12 个病灶类别的综合实验验证了 Malenia 的优越性能。
英文摘要：
Recent advancements in medical vision-language pre-training models have driven significant progress in zero-shot disease recognition. However, transferring image-level knowledge to pixel-level tasks, such as lesion segmentation in 3D CT scans, remains a critical challenge. Due to the complexity and variability of pathological visual characteristics, existing methods struggle to align fine-grained lesion features not encountered during training with disease-related textual representations. In this paper, we present Malenia, a novel multi-scale lesion-level mask-attribute alignment framework, specifically designed for 3D zero-shot lesion segmentation. Malenia improves the compatibility between mask representations and their associated elemental attributes, explicitly linking the visual features of unseen lesions with the extensible knowledge learned from previously seen ones. Furthermore, we design a Cross-Modal Knowledge Injection module to enhance both visual and textual features with mutually beneficial information, effectively guiding the generation of segmentation results. Comprehensive experiments across three datasets and 12 lesion categories validate the superior performance of Malenia.
National Institutes of Health Clinical Center（美国国立卫生研究院临床中心）；Ping An Insurance Company of China（中国平安保险（集团）股份有限公司）
Automatic lesion detection from computed tomography (CT) scans is an important task in medical imaging analysis. It is still very challenging due to similar appearances (e.g. intensity and texture) between lesions and other tissues, making it especially difficult to develop a universal lesion detector. Instead of developing a specific-type lesion detector, this work builds a Universal Lesion Detector (ULDor) based on Mask R-CNN, which is able to detect all different kinds of lesions from whole body parts. As a state-of-the-art object detector, Mask R-CNN adds a branch for predicting segmentati…
Privileged Lesion-Context Relational Distillation for Mask-Free Skin Lesion Classification
Victoria University（维多利亚大学）；Bangaldesh University of Business and Technology（孟加拉国商业与技术大学）；Daffodil Institute of Information Technology（达芙妮信息技术学院）；American International University-Bangladesh (AIUB)（孟加拉国美国国际大学）；Multimedia University（多媒体大学）
Huazhong University of Science & Technology（华中科技大学）；Tencent PCG（腾讯平台与内容事业群）
We present an approach to efficiently and effectively adapt a masked image modeling (MIM) pre-trained vanilla Vision Transformer (ViT) for object detection, which is based on our two novel observations: (i) A MIM pre-trained vanilla ViT encoder can work surprisingly well in the challenging object-level recognition scenario even with randomly sampled partial observations, e.g., only 25% $\sim$ 50% of the input embeddings. (ii) In order to construct multi-scale representations for object detection from single-scale ViT, a randomly initialized compact convolutional stem supplants the pre-trained …
Skin lesion segmentation is a vital task in skin cancer diagnosis and further treatment. Although deep learning based approaches have significantly improved the segmentation accuracy, these algorithms are still reliant on having a large enough dataset in order to achieve adequate results. Inspired by the immense success of generative adversarial networks (GANs), we propose a GAN-based augmentation of the original dataset in order to improve the segmentation performance. In particular, we use the segmentation masks available in the training dataset to train the Mask2Lesion model, and use the mo…
Tel Aviv University（特拉维夫大学）；Bar-Ilan University（巴伊兰大学）
This paper explores the use of a soft ground-truth mask ("soft mask'') to train a Fully Convolutional Neural Network (FCNN) for segmentation of Multiple Sclerosis (MS) lesions. Detection and segmentation of MS lesions is a complex task largely due to the extreme unbalanced data, with very small number of lesion pixels that can be used for training. Utilizing the anatomical knowledge that the lesion surrounding pixels may also include some lesion level information, we suggest to increase the data set of the lesion class with neighboring pixel data - with a reduced confidence weight. A soft mask…
Northwestern Polytechnical University（西北工业大学）；University of Adelaide（阿德莱德大学）
Automated skin lesion segmentation and classification are two most essential and related tasks in the computer-aided diagnosis of skin cancer. Despite their prevalence, deep learning models are usually designed for only one task, ignoring the potential benefits in jointly performing both tasks. In this paper, we propose the mutual bootstrapping deep convolutional neural networks (MB-DCNN) model for simultaneous skin lesion segmentation and classification. This model consists of a coarse segmentation network (coarse-SN), a mask-guided classification network (mask-CN), and an enhanced segmentati…
MammoTracker: Mask-Guided Lesion Tracking in Temporal Mammograms
Department of Electrical and Computer Engineering, Duke University, Durham, NC（电气与计算机工程系，杜克大学）；Department of Biomedical Engineering, Duke University, Durham, NC（生物医学工程系，杜克大学）；Department of Radiology, Duke University School of Medicine, Durham,NC（放射学系，杜克大学医学学院）；Departments Population Health Science and Mathematics, Duke University School of Medicine, Durham,NC（流行病学与数学系，杜克大学医学学院）
AMLP: Adjustable Masking Lesion Patches for Self-Supervised Medical Image Segmentation
State Key Laboratory of Reliability and Intelligence of Electrical Equipment, School of Health Sciences and Biomedical Engineering, Hebei University of Technology, Tianjin, China（河北工业大学可靠性与智能电气设备国家重点实验室，健康科学与生物医学工程学院，中国天津）；Department of Computer Science, University of Oxford, Oxford, United Kingdom（英国牛津大学计算机科学系）；institute of Logic and Computation, Vienna University of Technology, Vienna, Austria（奥地利技术大学逻辑与计算研究所）
Medical Faculty Mannheim（曼海姆医学院）；Heidelberg University（海德堡大学）；Tooploox
We propose a novel procedure to improve liver and lesion segmentation from CT scans for U-Net based models. Our method extends standard segmentation pipelines to focus on higher target recall or reduction of noisy false-positive predictions, boosting overall segmentation performance. To achieve this, we include segmentation errors into a new learning process appended to the main training setup, allowing the model to find features which explain away previous errors. We evaluate this on semantically distinct architectures: cascaded two- and three-dimensional as well as combined learning setups f…
Mask-Enhanced Segment Anything Model for Tumor Lesion Semantic Segmentation
Beihang University（北京航空航天大学）；Hangzhou Innovation Institute, Beihang University（北京航空航天大学杭州创新研究院）；Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；The Chinese University of Hong Kong（香港中文大学）；Sun Yat-sen University（中山大学）；Peking University Cancer Hospital & Institute（北京大学肿瘤医院暨研究所）；Nanjing University（南京大学）
Accurate, automated lesion detection in Computed Tomography (CT) is an important yet challenging task due to the large variation of lesion types, sizes, locations and appearances. Recent work on CT lesion detection employs two-stage region proposal based methods trained with centroid or bounding-box annotations. We propose a highly accurate and efficient one-stage lesion detector, by re-designing a RetinaNet to meet the particular challenges in medical imaging. Specifically, we optimize the anchor configurations using a differential evolution search algorithm. For training, we leverage the res…
School of AI, University of the Chinese Academy of Science（中国科学院大学人工智能学院）；Institute of Computing Technology, CAS（中国科学院计算技术研究所）；School of Electronic, Electrical and Communication Engineering, University of the Chinese Academy of Science（中国科学院大学电子电气与通信工程学院）
Universal Lesion Detection (ULD) in computed tomography plays an essential role in computer-aided diagnosis. Promising ULD results have been reported by coarse-to-fine two-stage detection approaches, but such two-stage ULD methods still suffer from issues like imbalance of positive v.s. negative anchors during object proposal and insufficient supervision problem during localization regression and classification of the region of interest (RoI) proposals. While leveraging pseudo segmentation masks such as bounding map (BM) can reduce the above issues to some degree, it is still an open problem t…
Medical University of Vienna（维也纳医科大学）；TissueGnostics GmbH（TissueGnostics有限公司）
Malignant melanoma (MM) is one of the deadliest types of skin cancer. Analysing dermatoscopic images plays an important role in the early detection of MM and other pigmented skin lesions. Among different computer-based methods, deep learning-based approaches and in particular convolutional neural networks have shown excellent classification and segmentation performances for dermatoscopic skin lesion images. These models can be trained end-to-end without requiring any hand-crafted features. However, the effect of using lesion segmentation information on classification performance has remained a…
Stavanger University Hospital（斯塔万格大学医院）；University of Stavanger（斯塔万格大学）
Masked Image Modelling (MIM) has been shown to be an efficient self-supervised learning (SSL) pre-training paradigm when paired with transformer architectures and in the presence of a large amount of unlabelled natural images. The combination of the difficulties in accessing and obtaining large amounts of labeled data and the availability of unlabelled data in the medical imaging domain makes MIM an interesting approach to advance deep learning (DL) applications based on 3D medical imaging data. Nevertheless, SSL and, in particular, MIM applications with medical imaging data are rather scarce …
Prostate cancer (PCa) is the most common cancer in men in the United States. Multiparametic magnetic resonance imaging (mp-MRI) has been explored by many researchers to targeted prostate biopsies and radiation therapy. However, assessment on mp-MRI can be subjective, development of computer-aided diagnosis systems to automatically delineate the prostate gland and the intraprostratic lesions (ILs) becomes important to facilitate with radiologists in clinical practice. In this paper, we first study the implementation of the Mask-RCNN model to segment the prostate and ILs. We trained and evaluate…
National Institutes of Health（美国国立卫生研究院）；University of Florida（佛罗里达大学）；Nvidia Corp（英伟达公司）；Ping An Insurance (Group) Company of China Ltd（中国平安保险（集团）股份有限公司）
Volumetric lesion segmentation via medical imaging is a powerful means to precisely assess multiple time-point lesion/tumor changes. Because manual 3D segmentation is prohibitively time consuming and requires radiological experience, current practices rely on an imprecise surrogate called response evaluation criteria in solid tumors (RECIST). Despite their coarseness, RECIST marks are commonly found in current hospital picture and archiving systems (PACS), meaning they can provide a potentially powerful, yet extraordinarily challenging, source of weak supervision for full 3D segmentation. Towa…
University of Georgia（佐治亚大学）；Florida International University（佛罗里达国际大学）
Medical Imaging is one of the growing fields in the world of computer vision. In this study, we aim to address the Diabetic Retinopathy (DR) problem as one of the open challenges in medical imaging. In this research, we propose a new lesion detection architecture, comprising of two sub-modules, which is an optimal solution to detect and find not only the type of lesions caused by DR, their corresponding bounding boxes, and their masks; but also the severity level of the overall case. Aside from traditional accuracy, we also use two popular evaluation criteria to evaluate the outputs of our mod…
This paper addresses the problem of identifying two main types of lesions - Exudates and Microaneurysms - caused by Diabetic Retinopathy (DR) in the eyes of diabetic patients. We make use of Convolutional Neural Networks (CNNs) and Transfer Learning to locate and generate high-quality segmentation mask for each instance of the lesion that can be found in the patients' fundus images. We create our normalized database out of e-ophtha EX and e-ophtha MA and tweak Mask R-CNN to detect small lesions. Moreover, we employ data augmentation and the pre-trained weights of ResNet101 to compensate for ou…
Microsoft（微软）；Cupertino High School（库比蒂诺高中）；Inglemoor High School（英格尔穆尔高中）
All datasets contain some biases, often unintentional, due to how they were acquired and annotated. These biases distort machine-learning models' performance, creating spurious correlations that the models can unfairly exploit, or, contrarily destroying clear correlations that the models could learn. With the popularity of deep learning models, automated skin lesion analysis is starting to play an essential role in the early detection of Melanoma. The ISIC Archive is one of the most used skin lesion sources to benchmark deep learning-based tools. Bissoto et al. experimented with different boun…
National Institutes of Health（美国国立卫生研究院）；University of Florida（佛罗里达大学）；Ping An Insurance (Group) Company of China, Ltd.（中国平安保险（集团）股份有限公司）
Volumetric lesion segmentation from computed tomography (CT) images is a powerful means to precisely assess multiple time-point lesion/tumor changes. However, because manual 3D segmentation is prohibitively time consuming, current practices rely on an imprecise surrogate called response evaluation criteria in solid tumors (RECIST). Despite their coarseness, RECIST markers are commonly found in current hospital picture and archiving systems (PACS), meaning they can provide a potentially powerful, yet extraordinarily challenging, source of weak supervision for full 3D segmentation. Toward this e…
SLIM-Diff: Shared Latent Image-Mask Diffusion with Lp loss for Data-Scarce Epilepsy FLAIR MRI
ITIS Software, University of M\'alaga, Spain Department of Computer Languages；Computer Science, University of M\'alaga, Spain IBIMA Plataforma BIONAND, Instituto de Investigaci\'on Biom\'edica de M\'alaga, Spain Department of Radiology, Hospital Regional Universitario de M\'alaga, Spain
Unleashing the Potential of SAM for Medical Adaptation via Hierarchical Decoding
East China Normal University（华东师范大学）；Stanford University（斯坦福大学）；Johns Hopkins University（约翰斯·霍普金斯大学）；University of Hong Kong（香港大学）；University of Florida（佛罗里达大学）；UC Santa Cruz（加州大学圣克鲁兹分校）
Semi-Supervised Bone Marrow Lesion Detection from Knee MRI Segmentation Using Mask Inpainting Models
Gordon Center for Medical Imaging, Harvard Medical School and Massachusetts General Hospital（哈佛医学院与麻省总医院戈登医学成像中心）；Boston University（波士顿大学）；Seidenberg School of CSIS, Pace University（佩斯大学塞登伯格计算机科学与信息系统学院）；Ewha Womans University（梨花女子大学）
Memorial Sloan Kettering Cancer Center（纪念斯隆-凯特琳癌症中心）；Indian Institute of Technology, Madras（印度理工学院马德拉斯分校）
Purpose: Lesion segmentation in medical imaging is key to evaluating treatment response. We have recently shown that reinforcement learning can be applied to radiological images for lesion localization. Furthermore, we demonstrated that reinforcement learning addresses important limitations of supervised deep learning; namely, it can eliminate the requirement for large amounts of annotated training data and can provide valuable intuition lacking in supervised approaches. However, we did not address the fundamental task of lesion/structure-of-interest segmentation. Here we introduce a method co…
King’s College London（伦敦国王学院）；National Hospital for Neurology and Neurosurgery（国家神经内科和神经外科医院）；Thomas Jefferson University（托马斯·杰斐逊大学）；University of Washington（华盛顿大学）
Laser interstitial thermal therapy (LITT) is a novel minimally invasive treatment that is used to ablate intracranial structures to treat mesial temporal lobe epilepsy (MTLE). Region of interest (ROI) segmentation before and after LITT would enable automated lesion quantification to objectively assess treatment efficacy. Deep learning techniques, such as convolutional neural networks (CNNs) are state-of-the-art solutions for ROI segmentation, but require large amounts of annotated data during the training. However, collecting large datasets from emerging treatments such as LITT is impractical.…
ConvNeXtv2 Fusion with Mask R-CNN for Automatic Region Based Coronary Artery Stenosis Detection for Disease Diagnosis
Nepal Applied Mathematics and Informatics Institute for research（尼泊尔应用数学与信息学研究所）；Fogsphere(Redev AI Ltd.)（Fogsphere（Redev AI有限公司））；University of Lausanne（洛桑大学）；University of Aberdeen（阿伯丁大学）
Deep-learning-based approaches for retinal lesion segmentation often require an abundant amount of precise pixel-wise annotated data. However, coarse annotations such as circles or ellipses for outlining the lesion area can be six times more efficient than pixel-level annotation. Therefore, this paper proposes an annotation refinement network to convert a coarse annotation into a pixel-level segmentation mask. Our main novelty is the application of the prototype learning paradigm to enhance the generalization ability across different datasets or types of lesions. We also introduce a prototype …
Imaging Biomarkers and Computer-Aided Diagnosis Laboratory, Radiology and Imaging Sciences, National Institutes of Health Clinical Center（美国国立卫生研究院临床中心放射与成像科学部成像生物标志物和计算机辅助诊断实验室）；Ping An Insurance Company of China（中国平安保险（集团）股份有限公司）
Lesion segmentation on computed tomography (CT) scans is an important step for precisely monitoring changes in lesion/tumor growth. This task, however, is very challenging since manual segmentation is prohibitively time-consuming, expensive, and requires professional knowledge. Current practices rely on an imprecise substitute called response evaluation criteria in solid tumors (RECIST). Although these markers lack detailed information about the lesion regions, they are commonly found in hospitals' picture archiving and communication systems (PACS). Thus, these markers have the potential to se…
Data-driven automatic approaches have demonstrated their great potential in resolving various clinical diagnostic dilemmas for patients with malignant gliomas in neuro-oncology with the help of conventional and advanced molecular MR images. However, the lack of sufficient annotated MRI data has vastly impeded the development of such automatic methods. Conventional data augmentation approaches, including flipping, scaling, rotation, and distortion are not capable of generating data with diverse image content. In this paper, we propose a method, called synthesis of anatomic and molecular MR imag…
Unleashing Diffusion and State Space Models for Medical Image Segmentation
Department of Biostatistics, School of Global Public Health, New York University（生物统计学系、全球公共卫生学院、纽约大学）；School of Statistics, KLATASDS-MOE, East China Normal University（统计学系、KLATASDS-MOE、华东师范大学）；School of Biomedical Engineering, Southern Medical University（生物医学工程学院、南方医科大学）；Faculty of Biomedical Engineering, Shenzhen University of Advanced Technology（生物医学工程学院、深圳先进技术大学）
Istanbul Technical University（伊斯坦布尔理工大学）；Acibadem Mehmet Ali Aydinlar University School of Medicine（阿吉巴德姆·穆罕默德·阿里·艾丁拉尔大学医学院）；Bahcesehir University（巴耶塞特大学）
Biparametric MRI has emerged as an alternative to multiparametric prostate MRI, which eliminates the need for the potential harms to the patient due to the contrast medium. One major issue with biparametric MRI is difficulty to detect clinically significant prostate cancer (csPCA). Deep learning algorithms have emerged as an alternative solution to detect csPCA in cohort studies. We present a workflow which predicts csPCA on biparametric prostate MRI PI-CAI 2022 Challenge with over 10,000 carefully-curated prostate MRI exams. We propose to to segment the prostate gland first to the central gla…
Tarbiat Modares University（塔比亚特·莫达雷斯大学）；NRP company（NRP公司）；University of Sheffield（谢菲尔德大学）
Lesion segmentation is the first step in most automatic melanoma recognition systems. Deficiencies and difficulties in dermoscopic images such as color inconstancy, hair occlusion, dark corners and color charts make lesion segmentation an intricate task. In order to detect the lesion in the presence of these problems, we propose a supervised saliency detection method tailored for dermoscopic images based on the discriminative regional feature integration (DRFI). DRFI method incorporates multi-level segmentation, regional contrast, property, background descriptors, and a random forest regressor…
GraphDerm: Fusing Imaging, Physical Scale, and Metadata in a Population-Graph Classifier for Dermoscopic Lesions
School of Computer Science, Institute for Research in Fundamental Sciences (IPM)（计算机科学学院，基础科学研究所（IPM））；Department of Physics, Shahid Beheshti University（物理系，肖德·贝赫谢什提大学）；Faculty of Science, University of Tehran（科学学院，德黑兰大学）；Fatemeh Zahra Hospital, Alborz University of Medical Sciences（法特梅·扎拉医院，阿尔bor兹医学科学大学）；Department of Electrical and Computer Engineering, Sungkyunkwan University（电气与计算机工程系，成均馆大学）；Department of Computer Science and Engineering, Shahid Beheshti University（计算机科学与工程系，肖德·贝赫谢什提大学）；Autoimmune Bullous Diseases Research Center, Razi Hospital, Tehran University of Medical Sciences（自身免疫性水疱病研究中心，拉齐医院，德黑兰医学科学大学）
PETAR: Localized Findings Generation with Mask-Aware Vision-Language Modeling for PET Automated Reporting
University of Wisconsin–Madison Department of Computer Sciences（威斯康星大学麦迪逊分校计算机科学系）；University of Wisconsin–Madison Department Radiology（威斯康星大学麦迪逊分校放射学系）；Microsoft（微软公司）
Saratoga High School（萨拉托加高中）；Stanford University（斯坦福大学）
Chest X-rays of coronavirus disease 2019 (COVID-19) patients are frequently obtained to determine the extent of lung disease and are a valuable source of data for creating artificial intelligence models. Most work to date assessing disease severity on chest imaging has focused on segmenting computed tomography (CT) images; however, given that CTs are performed much less frequently than chest X-rays for COVID-19 patients, automated lung lesion segmentation on chest X-rays could be clinically valuable. There currently exists a universal shortage of chest X-rays with ground truth COVID-19 lung le…
Data augmentation can effectively resolve a scarcity of images when training machine-learning algorithms. It can make them more robust to unseen images. We present a lesion conditional Generative Adversarial Network LcGAN to generate synthetic Computed Tomography (CT) images for data augmentation. A lesion conditional image (segmented mask) is an input to both the generator and the discriminator of the LcGAN during training. The trained model generates contextual CT images based on input masks. We quantify the quality of the images by using a fully convolutional network (FCN) score and blurrin…
Whiting School of Engineering, Johns Hopkins University（约翰斯·霍普金斯大学怀廷工程学院）；School of Medicine, Johns Hopkins University（约翰斯·霍普金斯大学医学院）
Data-driven automatic approaches have demonstrated their great potential in resolving various clinical diagnostic dilemmas in neuro-oncology, especially with the help of standard anatomic and advanced molecular MR images. However, data quantity and quality remain a key determinant of, and a significant limit on, the potential of such applications. In our previous work, we explored synthesis of anatomic and molecular MR image network (SAMR) in patients with post-treatment malignant glioms. Now, we extend it and propose Confidence Guided SAMR (CG-SAMR) that synthesizes data from lesion informati…
