The Segment Anything Model (SAM), originally built on a 2D Vision Transformer (ViT), excels at capturing global patterns in 2D natural images but struggles with 3D medical imaging modalities like CT and MRI. These modalities require capturing spatial information in volumetric space for tasks such as organ segmentation and tumor quantification. To address this challenge, we introduce RefSAM3D, which adapts SAM for 3D medical imaging by incorporating a 3D image adapter and cross-modal reference prompt generation. Our approach modifies the visual encoder to handle 3D inputs and enhances the mask decoder for direct 3D mask generation. We also integrate textual prompts to improve segmentation accuracy and consistency in complex anatomical scenarios. By employing a hierarchical attention mechanism, our model effectively captures and integrates information across different scales. Extensive evaluations on multiple medical imaging datasets demonstrate the superior performance of RefSAM3D over state-of-the-art methods. Our contributions advance the application of SAM in accurately segmenting complex anatomical structures in medical imaging.
MM-DINOv2: Adapting Foundation Models for Multi-Modal Medical Image Analysis
Chair for AI for Image-Guided Diagnosis and Therapy, Technical University of Munich (TUM)（人工智能辅助影像诊断与治疗研究所，慕尼黑技术大学）；TUM University Hospital, Munich, Germany（慕尼黑技术大学医院）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）；Chair for AI in Healthcare and Medicine, Technical University of Munich (TUM)（人工智能在医疗与健康领域研究所，慕尼黑技术大学）；Department of Radiation Oncology, TUM University Hospital, Munich, Germany（放射肿瘤科，慕尼黑技术大学医院）；Chair for Computer Vision and Artificial Intelligence, Technical University of Munich (TUM)（计算机视觉与人工智能研究所，慕尼黑技术大学）；Department of Scientific Computing, Florida State University（科学计算系，佛罗里达州立大学）；Deutsches Konsortium für Translationale Krebsforschung (DKTK), Partner Site Munich（德国转化癌症研究联盟（DKTK）慕尼黑分部）；Institute of Radiation Medicine (IRM), Department of Radiation Sciences (DRS), Helmholtz Center Munich（放射医学研究所（IRM），辐射科学部门（DRS），海德堡中心慕尼黑）；Institute for Advanced Study, Technical University of Munich (TUM)（高级研究所，慕尼黑技术大学）
Adapting Segment Anything Model 3 for Concept-Driven Lesion Segmentation in Medical Images: An Experimental Study
The Medical Artificial Intelligence and Automation (MAIA) Laboratory, Department of Radiation Oncology, University of Texas Southwestern Medical Center（德克萨斯大学西南医学中心放射肿瘤学系医学人工智能与自动化（MAIA）实验室）；Medical Image Processing Group, Department of Radiology, University of Pennsylvania（宾夕法尼亚大学放射学系医学图像处理组）；Division of Digestive and Liver Diseases, Department of Internal Medicine, University of Texas Southwestern Medical Center（德克萨斯大学西南医学中心内科学系消化与肝脏疾病科）
Cross-Modal MRI Ovary Segmentation in Endometriosis Using Unpaired TVUS Prototype Priors
Center for AI and Data Science (CAIDAS), Julius-Maximilians-Universität Würzburg（维尔茨堡大学人工智能与数据科学中心）；Institute for Computational Imaging and AI in Medicine (CompAI), Technical University of Munich（慕尼黑工业大学计算成像与医学人工智能研究所）；Pattern Recognition Lab, Friedrich-Alexander Universität Erlangen-Nürnberg（埃尔朗根-纽伦堡大学模式识别实验室）；Department Artificial Intelligence in Biomedical Engineering (AIBE), Friedrich-Alexander-Universität Erlangen-Nürnberg（埃尔朗根-纽伦堡大学生物医学工程人工智能系）
MedAtlas: Evaluating LLMs for Multi-Round, Multi-Task Medical Reasoning Across Diverse Imaging Modalities and Clinical Text
University of Science and Technology of China（中国科学技术大学）；Eastern Institute of Technology（东方理工大学）；Shanghai Jiao Tong University（上海交通大学）；Hefei Comprehensive National Science Center（合肥综合性国家科学中心）
Efficient Parameter Adaptation for Multi-Modal Medical Image Segmentation and Prognosis
Department of Computer Vision, Mohamed bin Zayed University of Artificial Intelligence（计算机视觉系，Mohamed bin Zayed人工智能大学）；Department of Machine Learning, Mohamed bin Zayed University of Artificial Intelligence（机器学习系，Mohamed bin Zayed人工智能大学）；Michigan State University（密歇根州立大学）；M42 Health（M42健康）
Adapting SAM 2 for Visual Object Tracking: 1st Place Solution for MMVPR Challenge Multi-Modal Tracking
University of Washington（华盛顿大学）；Electronics and Telecommunications Research Institute（电子电信研究所）；National Center for High-performance Computing（高性能计算国家中心）
SMC-UDA: Structure-Modal Constraint for Unsupervised Cross-Domain Renal Segmentation
Xidian University（西安电子科技大学）；Rhode Island Hospital（罗德岛医院）；Warren Alpert Medical School of Brown University（布朗大学沃伦·阿尔珀特医学院）；Central South University（中南大学）；Johns Hopkins University School of Medicine（约翰斯·霍普金斯大学医学院）
Center for Artificial Intelligence in Medicine and Imaging（医学与成像人工智能中心）；Stanford University（斯坦福大学）
Multi-modal foundation models are typically trained on millions of pairs of natural images and text captions, frequently obtained through web-crawling approaches. Although such models depict excellent generative capabilities, they do not typically generalize well to specific domains such as medical images that have fundamentally shifted distributions compared to natural images. Building generative models for medical images that faithfully depict clinical context may help alleviate the paucity of healthcare datasets. Thus, in this study, we seek to research and expand the representational capab…
Incorporating Clinical Guidelines through Adapting Multi-modal Large Language Model for Prostate Cancer PI-RADS Scoring
The Chinese University of Hong Kong（香港中文大学）；Shanghai Jiao Tong University（上海交通大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Technical University of Denmark（丹麦技术大学）
GrInAdapt: Scaling Retinal Vessel Structural Map Segmentation Through Grounding, Integrating and Adapting Multi-device, Multi-site, and Multi-modal Fundus Domains
Department of Bioengineering, University of Washington（华盛顿大学生物工程系）；Department of Electrical and Computer Engineering, University of Washington（华盛顿大学电气与计算机工程系）；Department of Ophthalmology, University of Washington（华盛顿大学眼科系）
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Peng Cheng Laboratory（鹏城实验室）；University of Chinese Academy of Sciences（中国科学院大学）
Although significant progress has been made in few-shot learning, most of existing few-shot image classification methods require supervised pre-training on a large amount of samples of base classes, which limits their generalization ability in real world application. Recently, large-scale Vision-Language Pre-trained models (VLPs) have been gaining increasing attention in few-shot learning because they can provide a new paradigm for transferable visual representation learning with easily available text on the Web. However, the VLPs may neglect detailed visual information that is difficult to de…
TU Wien（维也纳工业大学）；ÉTS（魁北克大学高等技术学院）；CentraleSupélec-CNRS-Université Paris Sud（中央理工-高等电力学院-法国国家科学研究中心-巴黎南大学）
Minimization of distribution matching losses is a principled approach to domain adaptation in the context of image classification. However, it is largely overlooked in adapting segmentation networks, which is currently dominated by adversarial models. We propose a class of loss functions, which encourage direct kernel density matching in the network-output space, up to some geometric transformations computed from unlabeled inputs. Rather than using an intermediate domain discriminator, our direct approach unifies distribution matching and segmentation in a single loss. Therefore, it simplifies…
Visual question answering (VQA) is the multi-modal task of answering natural language questions about an input image. Through cross-dataset adaptation methods, it is possible to transfer knowledge from a source dataset with larger train samples to a target dataset where training set is limited. Suppose a VQA model trained on one dataset train set fails in adapting to another, it is hard to identify the underlying cause of domain mismatch as there could exists a multitude of reasons such as image distribution mismatch and question distribution mismatch. At UCLA, we are working on a VQG module t…
Southeast University（东南大学）；JD Explore Academy（京东探索研究院）；National University of Singapore（新加坡国立大学）；University of Melbourne（墨尔本大学）
Relying on deep supervised or self-supervised learning, previous methods for depth completion from paired single image and sparse depth data have achieved impressive performance in recent years. However, facing a new environment where the test data occurs online and differs from the training data in the RGB image content and depth sparsity, the trained model might suffer severe performance drop. To encourage the trained model to work well in such conditions, we expect it to be capable of adapting to the new environment continuously and effectively. To achieve this, we propose MetaComp. It util…
Adapting 2D Multi-Modal Large Language Model for 3D CT Image Analysis
Department of Electronic and Computer Engineering, The Hong Kong University of Science and Technology（香港科技大学电子及计算机工程学系）；Department of Computer Science and Engineering, The Chinese University of Hong Kong（香港中文大学计算机科学与工程学系）；Institute of Medical Intelligence and XR, The Chinese University of Hong Kong（香港中文大学医学智能与扩展现实研究所）；Center for Artificial Intelligence and Robotics, Hong Kong Institute of Science and Innovation, Chinese Academy of Sciences（中国科学院香港创新研究院人工智能与机器人创新中心）
RefSAM: Efficiently Adapting Segmenting Anything Model for Referring Video Object Segmentation
College of Computer Science and Technology, National University of Defense Technology（国防科技大学计算机学院）；School of Computer Science, The University of Sydney（悉尼大学计算机学院）
German Cancer Research Center (DKFZ)（德国癌症研究中心（DKFZ））
The U-Net was presented in 2015. With its straight-forward and successful architecture it quickly evolved to a commonly used benchmark in medical image segmentation. The adaptation of the U-Net to novel problems, however, comprises several degrees of freedom regarding the exact architecture, preprocessing, training and inference. These choices are not independent of each other and substantially impact the overall performance. The present paper introduces the nnU-Net ('no-new-Net'), which refers to a robust and self-adapting framework on the basis of 2D and 3D vanilla U-Nets. We argue the stron…
Efficient Medical Vision-Language Alignment Through Adapting Masked Vision Models
School of Informatics, Xiamen University（厦门大学信息学院）；Center for Smart Health, School of Nursing, The Hong Kong Polytechnic University（香港理工大学护理学院智能健康中心）；Department of Biomedical Informatics, Harvard Medical School（哈佛医学院生物医学信息学系）；Department of Radiology, Zhongshan Hospital (Xiamen), Fudan University（复旦大学中山医院放射科）；National Institute for Data Science in Health and Medicine（医学与数据科学国家研究院）；Department of Computer Science, School of Informatics, Xiamen University（厦门大学信息学院计算机科学系）
DiscoVL: Unveiling Disentangled C ross-Modal Representation Learning via Orthogonal Adversarial Regularization for V ision-Language Models
Shandong Artificial Intelligence Institute（山东人工智能研究院）；Qilu University of Technology (Shandong Academy of Sciences)（齐鲁工业大学（山东省科学院））；University of Florida（佛罗里达大学）
The Chinese University of Hong Kong（香港中文大学）；University of Michigan（密歇根大学）；Imsight Medical Technology Inc.（Imsight医疗科技公司）
Convolutional networks (ConvNets) have achieved great successes in various challenging vision tasks. However, the performance of ConvNets would degrade when encountering the domain shift. The domain adaptation is more significant while challenging in the field of biomedical image analysis, where cross-modality data have largely different distributions. Given that annotating the medical data is especially expensive, the supervised transfer learning approaches are not quite optimal. In this paper, we propose an unsupervised domain adaptation framework with adversarial learning for cross-modality…
Instruction-ViT: Multi-Modal Prompts for Instruction Learning in ViT
University of Electronic Science and Technology of China（电子科技大学）；The University of Texas at Arlington（德克萨斯大学阿灵顿分校）；The University of Georgia（佐治亚大学）；Northwestern Polytechnical University（西北工业大学）；Chinese University of Hong Kong（香港中文大学）；Mayo Clinic（梅奥诊所）；Massachusetts General Hospital（麻省总医院）；Harvard Medical School（哈佛医学院）；ShanghaiTech University（上海科技大学）
SCING:Towards More Efficient and Robust Person Re-Identification through Selective Cross-modal Prompt Tuning
Huazhong University of Science and Technology（华中科技大学）；Huazhong Agricultural University（华中农业大学）；University of California, Santa Cruz（加州大学圣克ruz分校）；City University of Hong Kong (Dongguan)（香港城市大学（东莞））
University of Science and Technology of China（中国科学技术大学）；Renmin University of China（中国人民大学）；Microsoft Research（微软研究院）；University of Rochester（罗切斯特大学）
The pre-trained image-text models, like CLIP, have demonstrated the strong power of vision-language representation learned from a large scale of web-collected image-text data. In light of the well-learned visual features, some existing works transfer image representation to video domain and achieve good results. However, how to utilize image-language pre-trained model (e.g., CLIP) for video-language pre-training (post-pretraining) is still under explored. In this paper, we investigate two questions: 1) what are the factors hindering post-pretraining CLIP to further improve the performance on v…
Scalpel-SAM: A Semi-Supervised Paradigm for Adapting SAM to Infrared Small Object Detection
School of Computer Science, Chengdu University（成都大学计算机科学学院）；China University of Geosciences（中国地质大学）；Zhejiang University（浙江大学）；Intelligent Computing Infrastructure Innovation Center, Zhejiang Lab（浙江实验室智能计算基础设施创新中心）
ASLseg: Adapting SAM in the Loop for Semi-supervised Liver Tumor Segmentation
Southern University of Science and Technology（南方科技大学）；The University of Hong Kong（香港大学）；Jiaxing Research Institute, Southern University of Science and Technology（南方科技大学嘉兴研究院）
