Humans frequently use referring (identifying) expressions to refer to objects. Especially in ambiguous settings, humans prefer expressions (called relational referring expressions) that describe an object with respect to a distinguishing, unique object. Unlike studies on video object search using referring expressions, in this paper, our focus is on (i) relational referring expressions in highly ambiguous settings, and (ii) methods that can both generate and comprehend a referring expression. For this goal, we first introduce a new dataset for video object search with referring expressions that includes numerous copies of the objects, making it difficult to use non-relational expressions. Moreover, we train two baseline deep networks on this dataset, which show promising results. Finally, we propose a deep attention network that significantly outperforms the baselines on our dataset. The dataset and the codes are available at https://github.com/hazananayurt/viref.
The University of Western Australia（西澳大利亚大学）；Murdoch University（莫道克大学）；Sun Yat-sen University（中山大学）；The Hong Kong University of Science and Technology（香港科技大学）
KTH Royal Institute of Technology（瑞典皇家理工学院）；Middle East Technical University（中东技术大学）
Referring to objects in a natural and unambiguous manner is crucial for effective human-robot interaction. Previous research on learning-based referring expressions has focused primarily on comprehension tasks, while generating referring expressions is still mostly limited to rule-based methods. In this work, we propose a two-stage approach that relies on deep learning for estimating spatial relations to describe an object naturally and unambiguously with a referring expression. We compare our method to the state of the art algorithm in ambiguous environments (e.g., environments that include v…
Object grounding tasks aim to locate the target object in an image through verbal communications. Understanding human command is an important process needed for effective human-robot communication. However, this is challenging because human commands can be ambiguous and erroneous. This paper aims to disambiguate the human's referring expressions by allowing the agent to ask relevant questions based on semantic data obtained from scene graphs. We test if our agent can use relations between objects from a scene graph to ask semantically relevant questions that can disambiguate the original user …
Towards Robust Referring Video Object Segmentation with Cyclic Relational Consensus
Carnegie Mellon University（卡内基梅隆大学）；Microsoft Research Asia（微软亚洲研究院）；University of Michigan（密歇根大学）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）
MedGround: Bridging the Evidence Gap in Medical Vision-Language Models with Verified Grounding Data
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；DAMO Academy, Alibaba Group（阿里巴巴集团 DAMO 院）；Hupan Lab, Zhejiang Province（浙江省 Hupan 实验室）
Google Inc.（谷歌公司）；University of California, Los Angeles（加利福尼亚大学洛杉矶分校）；University of Oxford（牛津大学）；Johns Hopkins University（约翰斯·霍普金斯大学）
We propose a method that can generate an unambiguous description (known as a referring expression) of a specific object or region in an image, and which can also comprehend or interpret such an expression to infer which object is being described. We show that our method outperforms previous methods that generate descriptions of objects without taking into account other potentially ambiguous objects in the scene. Our model is inspired by recent successes of deep learning methods for image captioning, but while image captioning is difficult to evaluate, our task allows for easy objective evaluat…
Don't Guess, Just Ask: Resolving Ambiguity in Referring Segmentation via Multi-turn Clarification
School of Computer Science and Engineering, Sun Yat-sen University（中山大学计算机科学与工程学院）；Guangdong Province Key Laboratory of Information Security Technology（广东省信息安全技术重点实验室）；Key Laboratory of Machine Intelligence and Advanced Computing, Ministry of Education（教育部机器智能与先进计算重点实验室）
Many AI systems integrate sensor inputs, world knowledge, and human-provided information to perform inference. While such systems often treat the human input as flawless, humans are better thought of as hazy oracles whose input may be ambiguous or outside of the AI system's understanding. In such situations it makes sense for the AI system to defer its inference while it disambiguates the human-provided information by, for example, asking the human to rephrase the query. Though this approach has been considered in the past, current work is typically limited to application-specific methods and …
EviRCOD: Evidence-Guided Probabilistic Decoding for Referring Camouflaged Object Detection
West China Hospital, Sichuan University（四川大学华西医院）；Independent Researcher（独立研究员）；Department of Data Science and Artificial Intelligence, Auckland University of Technology（奥克兰理工大学数据科学与人工智能系）
GeoRefer-Bench: A Benchmark from Referring Pixels to Verifiable Geospatial Reasoning
Central South University（中南大学）；Jiangxi Normal University（江西师范大学）；Hohai University（河海大学）；Renmin University of China（中国人民大学）；The Chinese University of Hong Kong（香港中文大学）
Beihang University（北京航空航天大学）；Chinese Academy of Sciences（中国科学院）；University of Chinese Academy of Sciences（中国科学院大学）；Beijing Jiaotong University（北京交通大学）；Sun Yat-sen University（中山大学）；Pazhou Lab（琶洲实验室）
Given a natural language expression and an image/video, the goal of referring segmentation is to produce the pixel-level masks of the entities described by the subject of the expression. Previous approaches tackle this problem by implicit feature interaction and fusion between visual and linguistic modalities in a one-stage manner. However, human tends to solve the referring problem in a progressive manner based on informative words in the expression, i.e., first roughly locating candidate entities and then distinguishing the target one. In this paper, we propose a Cross-Modal Progressive Comp…
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）；School of Computer Science and Engineering, Beihang University（北京航空航天大学计算机学院）；Sun Yat-sen University（中山大学）；University of Technology Sydney（悉尼科技大学）；AI Institute（360人工智能研究院）
Referring image segmentation aims at segmenting the foreground masks of the entities that can well match the description given in the natural language expression. Previous approaches tackle this problem using implicit feature interaction and fusion between visual and linguistic modalities, but usually fail to explore informative words of the expression to well align features from the two modalities for accurately identifying the referred entity. In this paper, we propose a Cross-Modal Progressive Comprehension (CMPC) module and a Text-Guided Feature Exchange (TGFE) module to effectively addres…
GeoSelect: Spatial-Program Execution for Training-Free Referring Remote Sensing Image Segmentation
School of Internet, Anhui University（安徽大学互联网学院）；National Engineering Research Center for Agro-Ecological Big Data Analysis & Application, Anhui University（安徽大学农业生态大数据分析与应用国家工程研究中心）；State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University（武汉大学测绘遥感信息工程国家重点实验室）
Zhejiang University（浙江大学）；DAMO Academy, Alibaba Group（阿里巴巴集团达摩院）；Alibaba Group（阿里巴巴集团）；University of Cambridge（剑桥大学）；National University of Singapore（新加坡国立大学）
The task of Human-Object Interaction (HOI) detection targets fine-grained visual parsing of humans interacting with their environment, enabling a broad range of applications. Prior work has demonstrated the benefits of effective architecture design and integration of relevant cues for more accurate HOI detection. However, the design of an appropriate pre-training strategy for this task remains underexplored by existing approaches. To address this gap, we propose Relational Language-Image Pre-training (RLIP), a strategy for contrastive pre-training that leverages both entity and relation descri…
