In this paper, we are tackling the weakly-supervised referring expression grounding task, for the localization of a referent object in an image according to a query sentence, where the mapping between image regions and queries are not available during the training stage. In traditional methods, an object region that best matches the referring expression is picked out, and then the query sentence is reconstructed from the selected region, where the reconstruction difference serves as the loss for back-propagation. The existing methods, however, conduct both the matching and the reconstruction approximately as they ignore the fact that the matching correctness is unknown. To overcome this limitation, a discriminative triad is designed here as the basis to the solution, through which a query can be converted into one or multiple discriminative triads in a very scalable way. Based on the discriminative triad, we further propose the triad-level matching and reconstruction modules which are lightweight yet effective for the weakly-supervised training, making it three times lighter and faster than the previous state-of-the-art methods. One important merit of our work is its superior performance despite the simple and neat design. Specifically, the proposed method achieves a new state-of-the-art accuracy when evaluated on RefCOCO (39.21%), RefCOCO+ (39.18%) and RefCOCOg (43.24%) datasets, that is 4.17%, 4.08% and 7.8% higher than the previous one, respectively.
A-PAIR: A Benchmark and Identity-Consistent Grounding Framework for Air-Ground Cross-View Referring Person Detection
School of Automation, Southeast University(东南大学自动化学院);Faculty of Information Engineering and Automation, Kunming University of Science and Technology(昆明理工大学信息工程与自动化学院);School of Computer Science and Engineering, University of New South Wales(新南威尔士大学计算机科学与工程学院);School of Sports Training, Tianjin University of Sport(天津体育学院运动训练学院);Tianjin University(天津大学);National University of Defense Technology(国防科技大学)
Adaptive Reconstruction Network for Weakly Supervised Referring Expression Grounding
Key Lab of Intell. Info. Process., Inst. of Comput. Tech., CAS(中国科学院计算技术研究所智能信息处理重点实验室);University of Chinese Academy of Sciences(中国科学院大学);University of Science and Technology of China(中国科学技术大学);Peng Cheng Laboratory(鹏城实验室)
Hierarchical Alignment-enhanced Adaptive Grounding Network for Generalized Referring Expression Comprehension
School of Computer Science and Technology, Xi’an Jiaotong University(西安交通大学计算机科学与技术学院);Institute of Big Data, Fudan University(复旦大学大数据学院);Shanghai University of Finance and Economics(上海财经大学);Nanyang Technological University(南洋理工大学)
Real-Time Referring Expression Comprehension by Single-Stage Grounding Network
Tencent AI Lab(腾讯人工智能实验室);National University of Singapore(新加坡国立大学);University of Rochester(罗切斯特大学)
在本文中,我们提出了一种新的端到端的模型,即单级接地网络(SSG),在一个图像内的参考表达式的参考定位。不同于以往的多阶段模型,依赖于对象的建议或检测到的区域,我们提出的模型的目的是通过一个单一的阶段,而不诉诸区域的建议,以及随后的区域明智的特征提取理解所指的表达。具体地说,提出了一个多模态交互器,以集中地总结关于指称表达的局部区域特征。随后,提出了一个地面定位器,直接在给定的图像中定位所指的表达。为了进一步提高定位精度,提出了一种引导注意力机制,以强制接地集中在中心区域的指示。此外,通过利用和预测视觉属性信息,地面可以进一步区分图像中的所指对象,从而提高模型的性能。RefCOCO,RefCOCO+和RefCOCOg数据集上的实验表明,我们提出的SSG不依赖于任何区域建议,可以实现与其他先进模型相当的性能。此外,我们的SSG优于以前的模型,并在ReferItGame数据集上实现了最先进的性能。更重要的是,我们的SSG具有时间效率,可以在25 ms(每秒40个引用)内从RefCOCO数据集的416*416图像中获得引用表达式,平均使用Nvidia Tesla P40,比现有的多级模型加速超过9倍。
Entity-enhanced Adaptive Reconstruction Network for Weakly Supervised Referring Expression Grounding
University of Chinese Academy of Sciences(中国科学院大学);Peng Cheng Laboratory(鹏城实验室);University of Science and Technology of China(中国科学技术大学);Nanjing University of Science and Technology(南京理工大学);Huawei Cloud & AI(华为云与人工智能部门)
InstanceRefer: Cooperative Holistic Understanding for Visual Grounding on Point Clouds through Instance Multi-level Contextual Referring
The Chinese University of Hong Kong (Shenzhen)(香港中文大学(深圳));Shenzhen Research Institute of Big Data(深圳大数据研究院);Southern University of Science and Technology(南方科技大学)
AugRefer: Advancing 3D Visual Grounding via Cross-Modal Augmentation and Spatial Relation-based Referring
University of Science and Technology of China(中国科学技术大学);Singapore University of Technology and Design(新加坡科技设计大学);Hefei University of Technology(合肥工业大学)
MIMO: A medical vision language model with visual referring multimodal input and pixel grounding multimodal output
School of Software & Microelectronics, Peking University(软件与微电子学院,北京大学);School of Computer Science, Peking University(计算机学院,北京大学);National Engineering Research Center for Software Engineering, Peking University(软件工程国家工程研究中心,北京大学);Peking University Sixth Hospital(北京大学第六医院);Augusta University(奥古斯塔大学);Peking University First Hospital(北京大学第一医院)
Knowledge-guided Pairwise Reconstruction Network for Weakly Supervised Referring Expression Grounding
Institute of Computing Technology, CAS(中国科学院计算技术研究所);University of Chinese Academy of Sciences(中国科学院大学);University of Science and Technology of China(中国科学技术大学)
Enhancing Visual Grounding and Generalization: A Multi-Task Cycle Training Approach for Vision-Language Models
College of Electronics and Information Engineering, Tongji University(同济大学电子与信息工程学院);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Centre for Perceptual and Interactive Intelligence, the Chinese University of Hong Kong(香港中文大学感知与交互智能中心);Department of Electronic Engineering, the Chinese University of Hong Kong(香港中文大学电子工程系)
Towards Unified Referring Expression Segmentation Across Omni-Level Visual Target Granularities
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Beijing Academy of Artificial Intelligence(北京人工智能研究院);University of Science and Technology Beijing(北京科技大学);School of Computer Science and Technology, Beijing Jiaotong University(北京交通大学计算机与信息技术学院)
ReferFlow: Beyond Final-Target Localization for Complex Referring Expressions
State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University(西安交通大学人工智能与机器人研究所人机混合增强智能全国重点实验室);Xingchen AGI Lab, China Telecom Artificial Intelligence Technology (Beijing) Co., Ltd.(中国电信人工智能技术(北京)有限公司星辰AGI实验室);Beijing Normal University(北京师范大学);Zhejiang University(浙江大学);University of the Chinese Academy of Sciences(中国科学院大学);University of Science and Technology Beijing(北京科技大学);Shanghai Jiao Tong University(上海交通大学)
Unveiling Parts Beyond Objects:Towards Finer-Granularity Referring Expression Segmentation
Institute of Automation, Chinese Academy of Sciences (CASIA)(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences (UCAS)(中国科学院大学人工智能学院);Beijing Academy of Artificial Intelligence (BAAI)(北京人工智能研究院);University of Science and Technology Beijing (USTB)(北京科技大学)
MedGround: Bridging the Evidence Gap in Medical Vision-Language Models with Verified Grounding Data
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);DAMO Academy, Alibaba Group(阿里巴巴集团 DAMO 院);Hupan Lab, Zhejiang Province(浙江省 Hupan 实验室)
UniRef-UAV: A Multimodal Benchmark for Universal Referring in UAV Imagery
School of Automation, Northwestern Polytechnical University(自动化学院,西北工业大学);College of Missile Engineering, Rocket Force University of Engineering(导弹工程学院,火箭兵工程大学);School of Artificial Intelligence, Chongqing University of Posts and Telecommunications(人工智能学院,重庆邮电大学)
SynthRef: Generation of Synthetic Referring Expressions for Object Segmentation
Universitat Politècnica de Catalunya(加泰罗尼亚理工大学);National Technical University of Athens(雅典国立技术大学);Universitat Oberta de Catalunya(加泰罗尼亚开放大学);Barcelona Supercomputing Center(巴塞罗那超级计算中心);University of Stuttgart(斯图加特大学)
University of Science and Technology of China(中国科学技术大学);Nanyang Technological University(南洋理工大学);Hefei University of Technology(合肥工业大学);Singapore Management University(新加坡管理大学)