MEGVII Technology（旷视科技）；Inception Institute of Artificial Intelligence（起源人工智能研究院）；Beijing Academy of Artificial Intelligence（北京人工智能研究院）；Beijing Institute of Technology（北京理工大学）；University of Macau（澳门大学）
Existing referring understanding tasks tend to involve the detection of a single text-referred object. In this paper, we propose a new and general referring understanding task, termed referring multi-object tracking (RMOT). Its core idea is to employ a language expression as a semantic cue to guide the prediction of multi-object tracking. To the best of our knowledge, it is the first work to achieve an arbitrary number of referent object predictions in videos. To push forward RMOT, we construct one benchmark with scalable expressions based on KITTI, named Refer-KITTI. Specifically, it provides 18 videos with 818 expressions, and each expression in a video is annotated with an average of 10.7 objects. Further, we develop a transformer-based architecture TransRMOT to tackle the new task in an online manner, which achieves impressive detection performance and outperforms other counterparts. The dataset and code will be available at https://github.com/wudongming97/RMOT.
CD-RMOT-Bench: Benchmarking the Cross-Domain Referring Multi-Object Tracking
School of Computer Science and Technology, Tianjin University（天津大学计算机科学与技术学院）；Faculty of Computer Science and Artificial Intelligence, Shenzhen University of Advanced Technology（深圳先进技术大学计算机科学与人工智能学院）
Efficient Language-to-Vision Feature Injection for Referring Single-Object Tracking
Technology and Engineering Center for Space Utilization, Chinese Academy of Sciences（中国科学院空间应用工程与技术中心）；University of Chinese Academy of Sciences（中国科学院大学）
ORMOT: A Dataset and Framework for Omnidirectional Referring Multi-Object Tracking
State Key Laboratory of Multispectral Information Intelligent Processing Technology, School of Artificial Intelligence and Automation, Huazhong University of Science and Technology（多谱信息智能处理技术国家重点实验室，人工智能与自动化学院，华中科技大学）
Cognitive Disentanglement for Referring Multi-Object Tracking
Qingdao Institute of Software, College of Computer Science and Technology, China University of Petroleum (East China)（青岛软件研究所，计算机科学与技术学院，中国石油大学（华东））；Thrust of AI, Hong Kong University of Science and Technology (GuangZhou)（人工智能研究组，香港科学与技术大学（广州））；Shandong Key Laboratory of Intelligent Oil & Gas Industrial Software（山东省智能油气工业软件重点实验室）；Wangxuan Institute of Computer Technology, Peking University（王轩计算机技术研究所，北京大学）；School of Automation, Southeast University（自动化学院，东南大学）；School of Computer Science, Beijing Institute of Technology（计算机科学学院，北京理工大学）；School of Artificial Intelligence and Computer Science, Nantong University（人工智能与计算机科学学院，南通大学）；Thrust of Intelligent Transportation, Hong Kong University of Science and Technology (GuangZhou)（智能交通研究组，香港科学与技术大学（广州））；Department of Electrical Engineering and Electronics, University of Liverpool（电子工程与电子学院，利物浦大学）
COAL: Counterfactual and Observation-Enhanced Alignment Learning for Discriminative Referring Multi-Object Tracking
School of Automation, Southeast University, Nanjing, China（东南大学自动化学院，南京，中国）；Key Laboratory of Measurement and Control of Complex Systems of Engineering, Ministry of Education, Nanjing, China（工程复杂系统测量与控制国家重点实验室，教育部，南京，中国）；School of Physical & Mathematical Sciences, Nanyang Technological University, Singapore（南洋理工大学物理与数学科学学院，新加坡）；Big Data Institute, Central South University, Changsha, China（中南大学大数据研究院，长沙，中国）
Referring Multi-Object Tracking with Diverse Dynamic Expressions
School of Computer Science, National Engineering Research Center for Multimedia Software, Institute of Artificial Intelligence, Hubei Key Laboratory of Multimedia and Network Communication Engineering, Wuhan University（计算机科学学院、多媒体软件国家工程研究中心、人工智能研究院、湖北省多媒体与网络通信工程重点实验室、武汉大学）；Multimedia Lab (MMLab), Department of Information Engineering, the Chinese University of Hong Kong（多媒体实验室（MMLab）、信息工程系、香港中文大学）；State Key Laboratory of Internet of Things for Smart City, Department of Computer and Information Science, University of Macau（智能城市物联网国家重点实验室、计算机与信息科学系、澳门大学）；College of Information Technology, United Arab Emirates University（信息科技学院、阿联酋大学）
MEX: Memory-efficient Approach to Referring Multi-Object Tracking
University of Science（科学大学）；Vietnam National University, Ho Chi Minh City（越南国家大学，胡志明市）；Department of Computer Science and Computer Engineering（计算机科学与工程系）
Referring Video Object Segmentation with Cross-Modality Proxy Queries
DUT-RU International School of Information Science & Engineering, Dalian University of Technology, China（大连理工大学国际信息科学与工程学院，中国）；Chinese University of Hong Kong, China（香港中文大学，中国）；University of Sydney, Australia（悉尼大学，澳大利亚）
EchoTrack: Auditory Referring Multi-Object Tracking for Autonomous Driving
College of Computer Science and Electronic Engineering, Hunan University（湖南大学计算机与电子工程学院）；School of Robotics, Hunan University（湖南大学机器人学院）；National Engineering Research Center of Robot Visual Perception and Control Technology, Hunan University（湖南大学机器人视觉感知与控制技术国家工程研究中心）；Institute for Robotics and Anthropomatics, Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院机器人与人类工程学研究所）；Hangzhou SurImage Technology Company Ltd.（杭州 SurImage 科技有限公司）
Referring video object segmentation (R-VOS) is an emerging cross-modal task that aims to segment the target object referred by a language expression in all video frames. In this work, we propose a simple and unified framework built upon Transformer, termed ReferFormer. It views the language as queries and directly attends to the most relevant regions in the video frames. Concretely, we introduce a small set of object queries conditioned on the language as the input to the Transformer. In this manner, all the queries are obligated to find the referred objects only. They are eventually transform…
SVAC: Scaling Is All You Need For Referring Video Object Segmentation
Columbia University New York, USA（哥伦比亚大学）；Carnegie Mellon University Pittsburgh, USA（卡内基梅隆大学）；New York University New York, USA（纽约大学）；Wuhan University Wuhan, China（武汉大学）
Humans frequently use referring (identifying) expressions to refer to objects. Especially in ambiguous settings, humans prefer expressions (called relational referring expressions) that describe an object with respect to a distinguishing, unique object. Unlike studies on video object search using referring expressions, in this paper, our focus is on (i) relational referring expressions in highly ambiguous settings, and (ii) methods that can both generate and comprehend a referring expression. For this goal, we first introduce a new dataset for video object search with referring expressions tha…
School of Computer Science, Nankai University（南开大学计算机学院）；BNRist, Department of Computer Science and Technology, Tsinghua University（清华大学计算机科学与技术系BNRist（注：BNRist为北京信息科学与技术国家研究中心的英文缩写））
Academia Sinica（中央研究院）；NEC Laboratories America（美国NEC实验室）；University of California, Merced（加州大学默塞德分校）；Google Cloud（谷歌云）
Referring expressions are natural language descriptions that identify a particular object within a scene and are widely used in our daily conversations. In this work, we focus on segmenting the object in an image specified by a referring expression. To this end, we propose an end-to-end trainable comprehension network that consists of the language and visual encoders to extract feature representations from both domains. We introduce the spatial-aware dynamic filters to transfer knowledge from text to image, and effectively capture the spatial information of the specified object. To better comm…
Tomorrow Advancing Life (TAL) Education Group（好未来教育集团）
The task of referring video object segmentation aims to segment the object in the frames of a given video to which the referring expressions refer. Previous methods adopt multi-stage approach and design complex pipelines to obtain promising results. Recently, the end-to-end method based on Transformer has proved its superiority. In this work, we draw on the advantages of the above methods to provide a simple and effective pipeline for RVOS. Firstly, We improve the state-of-the-art one-stage method ReferFormer to obtain mask sequences that are strongly correlated with language descriptions. Sec…
EventRR: Event Referential Reasoning for Referring Video Object Segmentation
Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；The Hong Kong University of Science and Technology (Guangzhou)（香港科学与技术大学（广州））；The Hong Kong University of Science and Technology（香港科学与技术大学）；Northwestern Polytechnical University（西北工业大学）
The University of Hong Kong（香港大学）；Sun Yat-sen University（中山大学）；Deepwise AI Lab（深睿医疗人工智能实验室）
Referring expression comprehension aims to locate the object instance described by a natural language referring expression in an image. This task is compositional and inherently requires visual reasoning on top of the relationships among the objects in the image. Meanwhile, the visual reasoning process is guided by the linguistic structure of the referring expression. However, existing approaches treat the objects in isolation or only explore the first-order relationships between objects without being aligned with the potential complexity of the expression. Thus it is hard for them to adapt to…
We investigate the problem of object referring (OR) i.e. to localize a target object in a visual scene coming with a language description. Humans perceive the world more as continued video snippets than as static images, and describe objects not only by their appearance, but also by their spatio-temporal context and motion features. Humans also gaze at the object when they issue a referring expression. Existing works for OR mostly focus on static images only, which fall short in providing many such cues. This paper addresses OR in videos with language and human gaze. To that end, we present a …
Few-Shot Referring Video Single- and Multi-Object Segmentation via Cross-Modal Affinity with Instance Sequence Matching
School of Computer Science and Technology, Anhui University of Technology（安徽工业大学计算机科学与技术学院）；Department of Computer Science and Engineering, Southern University of Science and Technology（南方科技大学计算机科学与工程系）；School of Computer Science and Information Engineering, Hefei University of Technology（合肥工业大学计算机与信息工程学院）
The task of referring relationships is to localize subject and object entities in an image satisfying a relationship query, which is given in the form of \texttt{<subject, predicate, object>}. This requires simultaneous localization of the subject and object entities in a specified relationship. We introduce a simple yet effective proposal-based method for referring relationships. Different from the existing methods such as SSAS, our method can generate a high-resolution result while reducing its complexity and ambiguity. Our method is composed of two modules: a category-based proposal generat…
In this paper, we address the problem of referring expression comprehension in videos, which is challenging due to complex expression and scene dynamics. Unlike previous methods which solve the problem in multiple stages (i.e., tracking, proposal-based matching), we tackle the problem from a novel perspective, \textbf{co-grounding}, with an elegant one-stage framework. We enhance the single-frame grounding accuracy by semantic attention learning and improve the cross-frame grounding consistency with co-grounding feature learning. Semantic attention learning explicitly parses referring cues in …
The University of Adelaide（阿德莱德大学）；The University of Electronic Science and Technology of China（电子科技大学）
The task in referring expression comprehension is to localise the object instance in an image described by a referring expression phrased in natural language. As a language-to-vision matching task, the key to this problem is to learn a discriminative object feature that can adapt to the expression used. To avoid ambiguity, the expression normally tends to describe not only the properties of the referent itself, but also its relationships to its neighbourhood. To capture and exploit this important information we propose a graph-based, language-guided attention mechanism. Being composed of node …
Max Planck Institute for Informatics（马克斯·普朗克信息研究所）；Bosch Center for Artificial Intelligence（博世人工智能中心）；University of California, Berkeley（加利福尼亚大学伯克利分校）
Most state-of-the-art semi-supervised video object segmentation methods rely on a pixel-accurate mask of a target object provided for the first frame of a video. However, obtaining a detailed segmentation mask is expensive and time-consuming. In this work we explore an alternative way of identifying a target object, namely by employing language referring expressions. Besides being a more practical and natural way of pointing out a target object, using language specifications can help to avoid drift as well as make the system more robust to complex dynamics and appearance variations. Leveraging…
University of California Merced（加州大学默塞德分校）；NEC Labs America（NEC美国实验室）；Google Research（谷歌研究院）
Referring expression comprehension aims to localize objects identified by natural language descriptions. This is a challenging task as it requires understanding of both visual and language domains. One nature is that each object can be described by synonymous sentences with paraphrases, and such varieties in languages have critical impact on learning a comprehension model. While prior work usually treats each sentence and attends it to an object separately, we focus on learning a referring expression comprehension model that considers the property in synonymous sentences. To this end, we devel…
Progressive Prompt-Guided Cross-Modal Reasoning for Referring Image Segmentation
School of Computer Science and Artificial Intelligence, Wuhan University of Technology（武汉理工大学计算机科学与人工智能学院）；School of Artificial Intelligence and Automation, Huazhong University of Science and Technology（华中科技大学人工智能与自动化学院）；University of Electronic Science and Technology of China（电子科技大学）；Wuhan Vocational College of Software and Engineering（武汉软件工程职业学院）
