Most referring object detection (ROD) models, especially the modern grounding detectors, are designed for data-rich conditions, yet many practical deployments, such as robotics, augmented reality, and other specialized domains, would face severe label scarcity. In such regimes, end-to-end grounding detectors need to learn spatial and semantic structure from scratch, wasting precious samples. We ask a simple question: Can explicit reasoning priors help models learn more efficiently when data is scarce? To explore this, we first introduce a Data-efficient Referring Object Detection (De-ROD) task, which is a benchmark protocol for measuring ROD performance in low-data and few-shot settings. We then propose the HeROD (Heuristic-inspired ROD), a lightweight, model-agnostic framework that injects explicit, heuristic-inspired spatial and semantic reasoning priors, which are interpretable signals derived based on the referring phrase, into 3 stages of a modern DETR-style pipeline: proposal ranking, prediction fusion, and Hungarian matching. By biasing both training and inference toward plausible candidates, these priors promise to improve label efficiency and convergence performance. On RefCOCO, RefCOCO+, and RefCOCOg, HeROD consistently outperforms strong grounding baselines in scarce-label regimes. More broadly, our results suggest that integrating simple, interpretable reasoning priors provides a practical and extensible path toward better data-efficient vision-language understanding.
We investigate Referring Image Segmentation (RIS), which outputs a segmentation map corresponding to the natural language description. Addressing RIS efficiently requires considering the interactions happening across visual and linguistic modalities and the interactions within each modality. Existing methods are limited because they either compute different forms of interactions sequentially (leading to error propagation) or ignore intramodal interactions. We address this limitation by performing all three interactions simultaneously through a Synchronous Multi-Modal Fusion Module (SFM). Moreo…
Video-Instrument Synergistic Network for Referring Video Instrument Segmentation in Robotic Surgery
Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Imperial College London（帝国理工学院）；Hong Kong University of Science and Technology（香港科技大学）；The General Hospital of Southern Theatre Command of PLA（中国人民解放军南部战区总医院）；National University of Singapore（新加坡国立大学）
We propose a Vision-Language Transformer (VLT) framework for referring segmentation to facilitate deep interactions among multi-modal information and enhance the holistic understanding to vision-language features. There are different ways to understand the dynamic emphasis of a language expression, especially when interacting with the image. However, the learned queries in existing transformer works are fixed after training, which cannot cope with the randomness and huge diversity of the language expressions. To address this issue, we propose a Query Generation Module, which dynamically produc…
Progressive Prompt-Guided Cross-Modal Reasoning for Referring Image Segmentation
School of Computer Science and Artificial Intelligence, Wuhan University of Technology（武汉理工大学计算机科学与人工智能学院）；School of Artificial Intelligence and Automation, Huazhong University of Science and Technology（华中科技大学人工智能与自动化学院）；University of Electronic Science and Technology of China（电子科技大学）；Wuhan Vocational College of Software and Engineering（武汉软件工程职业学院）
University of Sydney（悉尼大学）；OPPO Research Institute（OPPO研究院）；Beijing University of Posts and Telecommunications（北京邮电大学）；Kuaishou Technology（快手科技）；University of Melbourne（墨尔本大学）
Referring image segmentation aims to segment a referent via a natural linguistic expression.Due to the distinct data properties between text and image, it is challenging for a network to well align text and pixel-level features. Existing approaches use pretrained models to facilitate learning, yet separately transfer the language/vision knowledge from pretrained models, ignoring the multi-modal corresponding information. Inspired by the recent advance in Contrastive Language-Image Pretraining (CLIP), in this paper, we propose an end-to-end CLIP-Driven Referring Image Segmentation framework (CR…
Beijing University of Posts and Telecommunications（北京邮电大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Beijing Academy of Artificial Intelligence（北京人工智能研究院）；Artificial Intelligence Research, Chinese Academy of Sciences（中国科学院人工智能研究院）；University of Chinese Academy of Sciences（中国科学院大学）
In this work, we address the task of referring image segmentation (RIS), which aims at predicting a segmentation mask for the object described by a natural language expression. Most existing methods focus on establishing unidirectional or directional relationships between visual and linguistic features to associate two modalities together, while the multi-scale context is ignored or insufficiently modeled. Multi-scale context is crucial to localize and segment those objects that have large scale variations during the multi-modal fusion process. To solve this problem, we propose a simple yet ef…
Bridging Vision and Language Encoders: Parameter-Efficient Tuning for Referring Image Segmentation
Sun Yat-sen University（中山大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Shenzhen Research Institute of Big Data（深圳大数据研究院）；AI Institute, Fudan University（复旦大学人工智能研究院）；Tencent AI Lab（腾讯人工智能实验室）
Technical University of Munich（慕尼黑工业大学）；Simon Fraser University（西蒙弗雷泽大学）
We introduce the task of dense captioning in 3D scans from commodity RGB-D sensors. As input, we assume a point cloud of a 3D scene; the expected output is the bounding boxes along with the descriptions for the underlying objects. To address the 3D object detection and description problems, we propose Scan2Cap, an end-to-end trained method, to detect objects in the input scene and describe them in natural language. We use an attention mechanism that generates descriptive tokens while referring to the related components in the local context. To reflect object relations (i.e. relative spatial re…
Referring Video Object Segmentation with Cross-Modality Proxy Queries
DUT-RU International School of Information Science & Engineering, Dalian University of Technology, China（大连理工大学国际信息科学与工程学院，中国）；Chinese University of Hong Kong, China（香港中文大学，中国）；University of Sydney, Australia（悉尼大学，澳大利亚）
This paper presents an overview and comparative analysis of our systems designed for the following two tracks in SAPIEN ManiSkill Challenge 2021: No Interaction Track: The No Interaction track targets for learning policies from pre-collected demonstration trajectories. We investigate both imitation learning-based approach, i.e., imitating the observed behavior using classical supervised learning techniques, and offline reinforcement learning-based approaches, for this track. Moreover, the geometry and texture structures of objects and robotic arms are exploited via Transformer-based networks t…
