Advancing Visual Grounding with Scene Knowledge: Benchmark and Method
The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Sun Yat-sen University(中山大学);Shenzhen Research Institute of Big Data(深圳大数据研究院);Tencent AI Lab(腾讯人工智能实验室);AI 3 Institute, Fudan University(复旦大学人工智能创新与产业研究院)
Zhihong Chen, Ruifei Zhang, Yibing Song, Xiang Wan, Guanbin Li
备注:
Computer Vision and Natural Language Processing. 21 pages, 14 figures. CVPR-2023
Visual grounding (VG) aims to establish fine-grained alignment between vision and language. Ideally, it can be a testbed for vision-and-language models to evaluate their understanding of the images and texts and their reasoning abilities over their joint space. However, most existing VG datasets are constructed using simple description texts, which do not require sufficient reasoning over the images and texts. This has been demonstrated in a recent study~\cite{luo2022goes}, where a simple LSTM-based text encoder without pretraining can achieve state-of-the-art performance on mainstream VG datasets. Therefore, in this paper, we propose a novel benchmark of \underline{S}cene \underline{K}nowledge-guided \underline{V}isual \underline{G}rounding (SK-VG), where the image content and referring expressions are not sufficient to ground the target objects, forcing the models to have a reasoning ability on the long-form scene knowledge. To perform this task, we propose two approaches to accept the triple-type input, where the former embeds knowledge into the image features before the image-query interaction; the latter leverages linguistic structure to assist in computing the image-text matching. We conduct extensive experiments to analyze the above methods and show that the proposed approaches achieve promising results but still leave room for improvement, including performance and interpretability. The dataset and code are available at \url{https://github.com/zhjohnchan/SK-VG}.
AugRefer: Advancing 3D Visual Grounding via Cross-Modal Augmentation and Spatial Relation-based Referring
University of Science and Technology of China(中国科学技术大学);Singapore University of Technology and Design(新加坡科技设计大学);Hefei University of Technology(合肥工业大学)
From Objects to Anywhere: A Holistic Benchmark for Multi-level Visual Grounding in 3D Scenes
State Key Laboratory of General Artificial Intelligence, BIGAI(通用人工智能国家重点实验室,BIGAI);Tsinghua University(清华大学);Peking University(北京大学);Beijing Institute of Technology(北京理工大学)
LSVG: Language-Guided Scene Graphs with 2D-Assisted Multi-Modal Encoding for 3D Visual Grounding
School of Automation Science and Engineering, South China University of Technology(自动化科学与工程学院,华南理工大学);School of Future Technology, South China University of Technology(未来技术学院,华南理工大学)
University of Science and Technology of China(中国科学技术大学);Nanyang Technological University(南洋理工大学);Hefei University of Technology(合肥工业大学);Singapore Management University(新加坡管理大学)
ASAP: Advancing Semantic Alignment Promotes Multi-Modal Manipulation Detecting and Grounding
School of Computer Science and Information Engineering, Hefei University of Technology, China(计算机科学与信息工程学院,合肥工业大学,中国);Institute of Artificial Intelligence, Hefei Comprehensive National Science Center, China(人工智能研究所,合肥综合性国家科学中心,中国)
Towards Unified Surgical Scene Understanding:Bridging Reasoning and Grounding via MLLMs
Southern University of Science and Technology(南方科技大学);Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences(深圳先进技术研究院,中国科学院);Northwestern University(西北大学);University of Alberta(阿尔伯塔大学);Yale University(耶鲁大学);Nanfang Hospital(南华医院);Shenzhen University of Advanced Technology(深圳大学先进技术研究院)
Advancing Complex Wide-Area Scene Understanding with Hierarchical Coresets Selection
Institute of Software Chinese Academy of Sciences(中国科学院软件研究所);University of the Chinese Academy of Sciences(中国科学院大学);Beijing University of Technology(北京理工大学)
Context Disentangling and Prototype Inheriting for Robust Visual Grounding
Nanjing University of Science and Technology(南京理工大学);Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);SenseTime Research(商汤科技研究院)
RGBT-GroundBench: Visual Grounding Beyond RGB in Complex Real-World Scenarios
Institute of Artificial Intelligence, State Key Laboratory of Virtual Reality Technology and Systems, Beihang University(北京航空航天大学人工智能研究院、虚拟现实技术与系统国家重点实验室);Pengcheng Laboratory(鹏城实验室);Shanghai Jiao Tong University(上海交通大学)
Move to Understand a 3D Scene: Bridging Visual Grounding and Exploration for Efficient and Versatile Embodied Navigation
Tsinghua University(清华大学);Beijing Institute of Technology(北京理工大学);Beihang University(北航);State Key Laboratory of General Artificial Intelligence, BIGAI, China(国家一般人工智能重点实验室, BIGAI, 中国)
ViewSRD: 3D Visual Grounding via Structured Multi-View Decomposition
South China University of Technology(华南理工大学);Guangdong Engineering Center for Large Model and GenAI Technology(广东省大模型与生成式人工智能技术工程中心);State Key Laboratory of Subtropical Building and Urban Science(亚热带建筑科学国家重点实验室);Ministry of Education Key Laboratory of Big Data and Intelligent Robot(教育部大数据与智能机器人重点实验室);Guangdong Provincial Key Lab of Computational Intelligence and Cyberspace Information(广东省计算智能与网络信息重点实验室);Singapore Management University(新加坡国立大学)
GeoWeaver: Grounding Visual Tokens with Geometric Evidence before Scene Reasoning
Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳));Pengcheng Laboratory(鹏城实验室);Pazhou Lab (Huangpu)(琶洲实验室(黄埔));Hainan University(海南大学);University of California at Merced(加州大学默塞德分校)