Tracking and segmentation play essential roles in video understanding, providing basic positional information and temporal association of objects within video sequences. Despite their shared objective, existing approaches often tackle these tasks using specialized architectures or modality-specific parameters, limiting their generalization and scalability. Recent efforts have attempted to unify multiple tracking and segmentation subtasks from the perspectives of any modality input or multi-task inference. However, these approaches tend to overlook two critical challenges: the distributional gap across different modalities and the feature representation gap across tasks. These issues hinder effective cross-task and cross-modal knowledge sharing, ultimately constraining the development of a true generalist model. To address these limitations, we propose a universal tracking and segmentation framework named SATA, which unifies a broad spectrum of tracking and segmentation subtasks with any modality input. Specifically, a Decoupled Mixture-of-Expert (DeMoE) mechanism is presented to decouple the unified representation learning task into the modeling process of cross-modal shared knowledge and specific information, thus enabling the model to maintain flexibility while enhancing generalization. Additionally, we introduce a Task-aware Multi-object Tracking (TaMOT) pipeline to unify all the task outputs as a unified set of instances with calibrated ID information, thereby alleviating the degradation of task-specific knowledge during multi-task training. SATA demonstrates superior performance on 18 challenging tracking and segmentation benchmarks, offering a novel perspective for more generalizable video understanding.
Detecting and segmenting individual objects, regardless of their category, is crucial for many applications such as action detection or robotic interaction. While this problem has been well-studied under the classic formulation of spatio-temporal grouping, state-of-the-art approaches do not make use of learning-based methods. To bridge this gap, we propose a simple learning-based approach for spatio-temporal grouping. Our approach leverages motion cues from optical flow as a bottom-up signal for separating objects from each other. Motion cues are then combined with appearance cues that provide…
RefSAM: Efficiently Adapting Segmenting Anything Model for Referring Video Object Segmentation
College of Computer Science and Technology, National University of Defense Technology（国防科技大学计算机学院）；School of Computer Science, The University of Sydney（悉尼大学计算机学院）
Multi-scale Contrastive Adaptor Learning for Segmenting Anything in Underperformed Scenes
School of Automation and Electrical Engineering, University of Science and Technology Beijing（北京科技大学自动化与电气工程学院）；Alibaba DAMO Academy（阿里巴巴达摩院）；College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）
Technical University of Denmark（丹麦技术大学）；Skåne University Hospital（斯科讷大学医院）；Danish Research Centre for Magnetic Resonance（丹麦磁共振研究中心）；Copenhagen University Hospital Hvidovre（哥本哈根大学医院维兹奥勒医院）；Copenhagen University Hospital Rigshospitalet（哥本哈根大学医院国立医院）；University College London（伦敦大学学院）；UCLH NHS Foundation Trust（伦敦大学学院医院NHS基金会信托）；National Hospital for Neurology and Neurosurgery（国立神经病学与神经外科医院）
In this paper we present a method for simultaneously segmenting brain tumors and an extensive set of organs-at-risk for radiation therapy planning of glioblastomas. The method combines a contrast-adaptive generative model for whole-brain segmentation with a new spatial regularization model of tumor shape using convolutional restricted Boltzmann machines. We demonstrate experimentally that the method is able to adapt to image acquisitions that differ substantially from any available training data, ensuring its applicability across treatment sites; that its tumor segmentation accuracy is compara…
Magnetic resonance images (MRIs) are widely used to quantify vestibular schwannoma and the cochlea. Recently, deep learning methods have shown state-of-the-art performance for segmenting these structures. However, training segmentation models may require manual labels in target domain, which is expensive and time-consuming. To overcome this problem, domain adaptation is an effective way to leverage information from source domain to obtain accurate segmentations without requiring manual labels in target domain. In this paper, we propose an unsupervised learning framework to segment the VS and c…
Huazhong University of Science and Technology（华中科技大学）；Kennesaw State University（肯尼索州立大学）
The crossMoDA challenge aims to automatically segment the vestibular schwannoma (VS) tumor and cochlea regions of unlabeled high-resolution T2 scans by leveraging labeled contrast-enhanced T1 scans. The 2022 edition extends the segmentation task by including multi-institutional scans. In this work, we proposed an unpaired cross-modality segmentation framework using data augmentation and hybrid convolutional networks. Considering heterogeneous distributions and various image sizes for multi-institutional scans, we apply the min-max normalization for scaling the intensities of all scans between …
Towards Segmenting the Invisible: An End-to-End Registration and Segmentation Framework for Weakly Supervised Tumour Analysis
Institute of Technical；Business Information Systems, Faculty of Computer Science, Otto von Guericke University Magdeburg, Magdeburg, Germany Human Technopole, Milan, Italy Institute of Medical Engineering, Faculty of Electrical Engineering；Information Technology, Otto von Guericke University Magdeburg, Magdeburg, Germany Centre for Behavioural Brain Sciences, Magdeburg, Germany
A lack of generalizability is one key limitation of deep learning based segmentation. Typically, one manually labels new training images when segmenting organs in different imaging modalities or segmenting abnormal organs from distinct disease cohorts. The manual efforts can be alleviated if one is able to reuse manual labels from one modality (e.g., MRI) to train a segmentation network for a new modality (e.g., CT). Previously, two stage methods have been proposed to use cycle generative adversarial networks (CycleGAN) to synthesize training images for a target modality. Then, these efforts t…
University of Florida（佛罗里达大学）；Medical Imaging Technologies, Siemens Healthcare（西门子医疗医学影像技术部门）
Synthesized medical images have several important applications, e.g., as an intermedium in cross-modality image registration and as supplementary training samples to boost the generalization capability of a classifier. Especially, synthesized computed tomography (CT) data can provide X-ray attenuation map for radiation therapy planning. In this work, we propose a generic cross-modality synthesis approach with the following targets: 1) synthesizing realistic looking 3D images using unpaired training data, 2) ensuring consistent anatomical structures, which could be changed by geometric distorti…
ViPSAM: Visual Prompting Medical Image Segmentation Using Segment Anything Model
Department of Artificial Intelligence, Sungkyunkwan University, Republic of Korea（人工智能系，成均馆大学，大韩民国）；Department of Radiation Oncology, Samsung Medical Center, Sungkyunkwan University School of Medicine, Republic of Korea（放射肿瘤科，三星医疗中心，成均馆大学医学院，大韩民国）；Department of MetaBioHealth, Sungkyunkwan University, Republic of Korea（元生物健康系，成均馆大学，大韩民国）
CUHK（香港中文大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Peking University（北京大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）
SAM3D: Segment Anything Model in Volumetric Medical Images
University of Arkansas（阿肯色大学）；University of Science, Vietnam National University（越南国家大学科学大学）；John von Neumann Institute, Vietnam National University（越南国家大学约翰·冯·诺依曼研究所）；West Virginia University（西弗吉尼亚大学）；University of Arkansas for Medical Sciences（阿肯色医科大学）
SAGD: Boundary-Enhanced Segment Anything in 3D Gaussian via Gaussian Decomposition
The Hong Kong Polytechnic University（香港理工大学）；Center for Artificial Intelligence and Robotics, HKISI, CAS（中国科学院香港创新研究院人工智能与机器人中心）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；University of Science and Technology Beijing（北京科技大学）；Shandong University（山东大学）
Prompting Segment Anything Model with Domain-Adaptive Prototype for Generalizable Medical Image Segmentation
National Engineering Research Center for Multimedia Software（国家多媒体软件工程技术研究中心）；Medical Artificial Intelligence Research Institute of Renmin Hospital, Wuhan University（武汉大学人民医院医学人工智能研究院）；School of Computer Science, Central China Normal University（华中师范大学计算机学院）
Quantifying the Limits of Segmentation Foundation Models: Modeling Challenges in Segmenting Tree-Like and Low-Contrast Objects
Department of Electrical and Computer Engineering（电气与计算机工程系）；Department of Computer Science（计算机科学系）；Department of Radiology（放射学系）；Department of Biostatistics & Bioinformatics（生物统计学与生物信息学系）；Minnesota Health Solutions（明尼苏达健康解决方案）
RadSAM: Segmenting 3D radiological images with a 2D promptable model
Raidium；Université de Paris Cité（巴黎大学）；AP-HP（法国国家卫生与医学研究院）；Hôpital Européen Georges Pompidou（欧仁·蓬皮杜医院）；Department of Vascular and Oncological Interventional Radiology（血管和肿瘤介入放射科）；HEKA INRIA；INSERM PARCC U 970（法国国家医学研究院 PARCC U 970）
Segment Anything in Pathology Images with Natural Language
Department of Computer Science and Engineering（计算机科学与工程系）；The Hong Kong University of Science and Technology（香港科学与技术大学）；School of Electronic Engineering and Information Science（电子工程与信息科学学院）；University of Science and Technology of China（中国科学技术大学）；The Chinese University of Hong Kong（香港中文大学）；Tencent AI Platform Department（腾讯人工智能平台部门）；Department of Anatomical and Cellular Pathology（解剖与细胞病理学部）；Department of Chemical and Biological Engineering（化学与生物工程系）；Division of Life Science（生命科学系）；HKUST Shenzhen-Hong Kong Collaborative Innovation Research Institute（香港科技大学深圳-香港协同创新研究院）；State Key Laboratory of Nervous System Disorders（神经系统疾病国家重点实验室）
Southern University of Science and Technology（南方科技大学）；The University of Hong Kong（香港大学）
Multi-modal magnetic resonance (MR) imaging provides great potential for diagnosing and analyzing brain gliomas. In clinical scenarios, common MR sequences such as T1, T2 and FLAIR can be obtained simultaneously in a single scanning process. However, acquiring contrast enhanced modalities such as T1ce requires additional time, cost, and injection of contrast agent. As such, it is clinically meaningful to develop a method to synthesize unavailable modalities which can also be used as additional inputs to downstream tasks (e.g., brain tumor segmentation) for performance enhancing. In this work, …
Shanghai Jiao Tong University（上海交通大学）；Shandong Cancer Hospital Affiliated to Shandong University（山东大学附属山东省肿瘤医院）；Deepwise AI Lab（深睿医疗人工智能实验室）
Segmentation of nasopharyngeal carcinoma (NPC) from Magnetic Resonance Images (MRI) is a crucial prerequisite for NPC radiotherapy. However, manually segmenting of NPC is time-consuming and labor-intensive. Additionally, single-modality MRI generally cannot provide enough information for its accurate delineation. Therefore, a multi-modality MRI fusion network (MMFNet) based on three modalities of MRI (T1, T2 and contrast-enhanced T1) is proposed to complete accurate segmentation of NPC. The backbone of MMFNet is designed as a multi-encoder-based network, consisting of several encoders to captu…
