Referring video object segmentation aims to segment and track a target object in a video using a natural language prompt. Existing methods typically fuse visual and textual features in a highly entangled manner, processing multi-modal information together to generate per-frame masks. However, this approach often struggles with ambiguous target identification, particularly in scenes with multiple similar objects, and fails to ensure consistent mask propagation across frames. To address these limitations, we introduce FindTrack, an efficient decoupled framework that separates target identification from mask propagation. FindTrack first adaptively selects a key frame by balancing segmentation confidence and vision-text alignment, establishing a robust reference for the target object. This reference is then utilized by a dedicated propagation module to track and segment the object across the entire video. By decoupling these processes, FindTrack effectively reduces ambiguities in target association and enhances segmentation consistency. FindTrack significantly outperforms all existing methods on public benchmarks, demonstrating its superiority.
MEGVII Technology（旷视科技）；Inception Institute of Artificial Intelligence（起源人工智能研究院）；Beijing Academy of Artificial Intelligence（北京人工智能研究院）；Beijing Institute of Technology（北京理工大学）；University of Macau（澳门大学）
Existing referring understanding tasks tend to involve the detection of a single text-referred object. In this paper, we propose a new and general referring understanding task, termed referring multi-object tracking (RMOT). Its core idea is to employ a language expression as a semantic cue to guide the prediction of multi-object tracking. To the best of our knowledge, it is the first work to achieve an arbitrary number of referent object predictions in videos. To push forward RMOT, we construct one benchmark with scalable expressions based on KITTI, named Refer-KITTI. Specifically, it provides…
CD-RMOT-Bench: Benchmarking the Cross-Domain Referring Multi-Object Tracking
School of Computer Science and Technology, Tianjin University（天津大学计算机科学与技术学院）；Faculty of Computer Science and Artificial Intelligence, Shenzhen University of Advanced Technology（深圳先进技术大学计算机科学与人工智能学院）
Efficient Language-to-Vision Feature Injection for Referring Single-Object Tracking
Technology and Engineering Center for Space Utilization, Chinese Academy of Sciences（中国科学院空间应用工程与技术中心）；University of Chinese Academy of Sciences（中国科学院大学）
ORMOT: A Dataset and Framework for Omnidirectional Referring Multi-Object Tracking
State Key Laboratory of Multispectral Information Intelligent Processing Technology, School of Artificial Intelligence and Automation, Huazhong University of Science and Technology（多谱信息智能处理技术国家重点实验室，人工智能与自动化学院，华中科技大学）
Cognitive Disentanglement for Referring Multi-Object Tracking
Qingdao Institute of Software, College of Computer Science and Technology, China University of Petroleum (East China)（青岛软件研究所，计算机科学与技术学院，中国石油大学（华东））；Thrust of AI, Hong Kong University of Science and Technology (GuangZhou)（人工智能研究组，香港科学与技术大学（广州））；Shandong Key Laboratory of Intelligent Oil & Gas Industrial Software（山东省智能油气工业软件重点实验室）；Wangxuan Institute of Computer Technology, Peking University（王轩计算机技术研究所，北京大学）；School of Automation, Southeast University（自动化学院，东南大学）；School of Computer Science, Beijing Institute of Technology（计算机科学学院，北京理工大学）；School of Artificial Intelligence and Computer Science, Nantong University（人工智能与计算机科学学院，南通大学）；Thrust of Intelligent Transportation, Hong Kong University of Science and Technology (GuangZhou)（智能交通研究组，香港科学与技术大学（广州））；Department of Electrical Engineering and Electronics, University of Liverpool（电子工程与电子学院，利物浦大学）
TrackRef3D: Multi-View Consistent Track-then-Label for Open-World Referring Segmentation in 3D Gaussian Splatting
East China Normal University, Shanghai, China（华东师范大学，上海，中国）；Shanghai AI Laboratory（上海人工智能实验室）；University of Electronic Science and Technology of China, Chengdu, China（电子科技大学，成都，中国）
Ant Group（蚂蚁集团）；Northwestern Polytechnical University（西北工业大学）；Tongji University（同济大学）
The referring video object segmentation task (RVOS) aims to segment object instances in a given video referred by a language expression in all video frames. Due to the requirement of understanding cross-modal semantics within individual instances, this task is more challenging than the traditional semi-supervised video object segmentation where the ground truth object masks in the first frame are given. With the great achievement of Transformer in object detection and object segmentation, RVOS has been made remarkable progress where ReferFormer achieved the state-of-the-art performance. In thi…
COAL: Counterfactual and Observation-Enhanced Alignment Learning for Discriminative Referring Multi-Object Tracking
School of Automation, Southeast University, Nanjing, China（东南大学自动化学院，南京，中国）；Key Laboratory of Measurement and Control of Complex Systems of Engineering, Ministry of Education, Nanjing, China（工程复杂系统测量与控制国家重点实验室，教育部，南京，中国）；School of Physical & Mathematical Sciences, Nanyang Technological University, Singapore（南洋理工大学物理与数学科学学院，新加坡）；Big Data Institute, Central South University, Changsha, China（中南大学大数据研究院，长沙，中国）
Referring image segmentation aims to segment the objects referred by a natural language expression. Previous methods usually focus on designing an implicit and recurrent feature interaction mechanism to fuse the visual-linguistic features to directly generate the final segmentation mask without explicitly modeling the localization information of the referent instances. To tackle these problems, we view this task from another perspective by decoupling it into a "Locate-Then-Segment" (LTS) scheme. Given a language expression, people generally first perform attention to the corresponding target i…
MEX: Memory-efficient Approach to Referring Multi-Object Tracking
University of Science（科学大学）；Vietnam National University, Ho Chi Minh City（越南国家大学，胡志明市）；Department of Computer Science and Computer Engineering（计算机科学与工程系）
Referring Multi-Object Tracking with Diverse Dynamic Expressions
School of Computer Science, National Engineering Research Center for Multimedia Software, Institute of Artificial Intelligence, Hubei Key Laboratory of Multimedia and Network Communication Engineering, Wuhan University（计算机科学学院、多媒体软件国家工程研究中心、人工智能研究院、湖北省多媒体与网络通信工程重点实验室、武汉大学）；Multimedia Lab (MMLab), Department of Information Engineering, the Chinese University of Hong Kong（多媒体实验室（MMLab）、信息工程系、香港中文大学）；State Key Laboratory of Internet of Things for Smart City, Department of Computer and Information Science, University of Macau（智能城市物联网国家重点实验室、计算机与信息科学系、澳门大学）；College of Information Technology, United Arab Emirates University（信息科技学院、阿联酋大学）
In this paper, we address the problem of referring expression comprehension in videos, which is challenging due to complex expression and scene dynamics. Unlike previous methods which solve the problem in multiple stages (i.e., tracking, proposal-based matching), we tackle the problem from a novel perspective, \textbf{co-grounding}, with an elegant one-stage framework. We enhance the single-frame grounding accuracy by semantic attention learning and improve the cross-frame grounding consistency with co-grounding feature learning. Semantic attention learning explicitly parses referring cues in …
Referring Video Object Segmentation with Cross-Modality Proxy Queries
DUT-RU International School of Information Science & Engineering, Dalian University of Technology, China（大连理工大学国际信息科学与工程学院，中国）；Chinese University of Hong Kong, China（香港中文大学，中国）；University of Sydney, Australia（悉尼大学，澳大利亚）
EchoTrack: Auditory Referring Multi-Object Tracking for Autonomous Driving
College of Computer Science and Electronic Engineering, Hunan University（湖南大学计算机与电子工程学院）；School of Robotics, Hunan University（湖南大学机器人学院）；National Engineering Research Center of Robot Visual Perception and Control Technology, Hunan University（湖南大学机器人视觉感知与控制技术国家工程研究中心）；Institute for Robotics and Anthropomatics, Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院机器人与人类工程学研究所）；Hangzhou SurImage Technology Company Ltd.（杭州 SurImage 科技有限公司）
RefSAM: Efficiently Adapting Segmenting Anything Model for Referring Video Object Segmentation
College of Computer Science and Technology, National University of Defense Technology（国防科技大学计算机学院）；School of Computer Science, The University of Sydney（悉尼大学计算机学院）
SetCon: Towards Open-Ended Referring Segmentation via Set-Level Concept Prediction
Shanghai Jiao Tong University（上海交通大学）；Shanghai Innovation Institute（上海创新研究院）；The Chinese University of Hong Kong（香港中文大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Fudan University（复旦大学）；University of Science and Technology of China（中国科学技术大学）
CoHD: A Counting-Aware Hierarchical Decoding Framework for Generalized Referring Expression Segmentation
Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；School of EIC, Huazhong University of Science and Technology（华中科技大学电子信息与工程学院）；Tencent TEG（腾讯技术工程事业群）
Referring video object segmentation (R-VOS) is an emerging cross-modal task that aims to segment the target object referred by a language expression in all video frames. In this work, we propose a simple and unified framework built upon Transformer, termed ReferFormer. It views the language as queries and directly attends to the most relevant regions in the video frames. Concretely, we introduce a small set of object queries conditioned on the language as the input to the Transformer. In this manner, all the queries are obligated to find the referred objects only. They are eventually transform…
