Grounding a command to the visual environment is an essential ingredient for interactions between autonomous vehicles and humans. In this work, we study the problem of language grounding for autonomous vehicles, which aims to localize a region in a visual scene according to a natural language command from a passenger. Prior work only employs the top layer representations of a vision-and-language pre-trained model to predict the region referred to by the command. However, such a method omits the useful features encoded in other layers, and thus results in inadequate understanding of the input scene and command. To tackle this limitation, we present the first layer fusion approach for this task. Since different visual regions may require distinct types of features to disambiguate them from each other, we further propose the region-specific dynamic (RSD) layer attention to adaptively fuse the multimodal information across layers for each region. Extensive experiments on the Talk2Car benchmark demonstrate that our approach helps predict more accurate regions and outperforms state-of-the-art methods.
视觉基础的任务需要在给定自然语言查询的情况下定位图像中最相关的区域或对象。到目前为止,这项任务的进展主要是在策划的数据集上衡量的,这些数据集并不总是代表人类口语。在这项工作中,我们偏离了最近流行的任务设置,并考虑在自动驾驶汽车的情况下的问题。特别是,我们考虑的情况下,乘客可以给自由形式的自然语言命令的车辆,可以与在街道场景中的对象。为了促进对这一主题的研究,我们根据最近的\emph{Talk2Car}数据集(URL:https://www.aicrowd.com/challenges/eccv-2020-commands-4-martous-vehicles)组织了\emph{Commands for Autonomous Vehicles}(C4 AV)挑战赛。本文介绍了挑战的结果。首先,我们将使用的基准与现有的数据集进行比较,以获得视觉基础。其次,我们确定了使表现最好的模型成功的方面,并将它们与现有的最先进的模型进行视觉基础,此外还通过对精心选择的子集进行评估来检测潜在的失败案例。最后,我们讨论了未来工作的几种可能性。
DDVT: Dynamic Dual-level Vision Transformer Fusion Network for Answer Grounding in Visual Question Answering
National and Local Joint Engineering Laboratory of Computer Aided Design, School of Software Engineering, Dalian University(大连大学软件工程学院计算机辅助设计国家地方联合工程实验室);School of Computer Science and Technology, Dalian University of Technology(大连理工大学计算机科学与技术学院)
Trifuse: Enhancing Attention-Based GUI Grounding via Multimodal Fusion
College of Computer Science and Technology, National University of Defense Technology(计算机科学与技术学院,国防科技大学);Intelligent Game and Decision Lab, Academy of Military Sciences(智能游戏与决策实验室,军事科学院)
TransVG++: End-to-End Visual Grounding with Language Conditioned Vision Transformer
University of Science and Technology of China(中国科学技术大学);Institute of Artificial Intelligence, Hefei Comprehensive National Science Center(合肥综合性国家科学中心人工智能研究院);Microsoft(微软);JD Explore Academy(京东探索研究院);Amazon(亚马逊);University of Sydney(悉尼大学);Shanghai Artificial Intelligent Laboratory(上海人工智能实验室)
Hand Trajectory Fusion for Egocentric Natural Language Query Grounding
Grupo de Tratamiento de Imágenes (GTI), Information Processing and Telecommunications Center , ETSI Telecomunicación, Universidad Politécnica de Madrid, Spain(图像处理小组(GTI)、信息处理与电信中心、电信工程学院、马德里理工大学、西班牙)
Audio-3DVG: Unified Audio -- Point Cloud Fusion for 3D Visual Grounding
Hanyang University(汉阳大学);University of Toronto(多伦多大学);University Health Network(大学健康网络);Knovel Engineering Lab(Knovel工程实验室);Michigan State University(密歇根州立大学);KU Leuven(鲁汶大学);German Research Center for Artificial Intelligence (DFKI)(德国人工智能研究中心);Max Planck Research School for Intelligent Systems (IMPRS-IS)(马克斯·普朗克智能系统研究学校);University of Stuttgart(斯图加特大学);UC Berkeley(伯克利大学);Johns Hopkins University(约翰霍普金斯大学)
TransVG: End-to-End Visual Grounding with Transformers
CAS Key Laboratory of GIPAS, University of Science and Technology of China(中国科学技术大学中国科学院GIPAS重点实验室);University of Rochester(罗切斯特大学);Institute of Artificial Intelligence, Hefei Comprehensive National Science Center(合肥综合性国家科学中心人工智能研究院)
Think Before You Drive: World Model-Inspired Multimodal Grounding for Autonomous Vehicles
University of Macau(澳门大学);UESTC(电子科技大学);Purdue University(普渡大学);McGill University(麦吉尔大学);Massachusetts Institute of Technology(麻省理工学院);University of Washington(华盛顿大学);The Hong Kong University of Science and Technology(香港科学与技术大学)
Talk2PC: Enhancing 3D Visual Grounding through LiDAR and Radar Point Clouds Fusion for Autonomous Driving
Thrust of Artificial Intelligence, Hong Kong University of Science and Technology (Guangzhou), China(香港理工大学(广州)人工智能研究所);Mononai AI, Sweden(蒙诺人工智能);College of Computer Science and Technology, China University of Petroleum (East China)(中国石油大学(华东)计算机科学与技术学院);School of Advanced Technology, Xi’an Jiaotong-Liverpool University(西安交通大学利物浦大学先进科技学院);Wangxuan Institute of Computer Technology, Peking University(北京大学王轩计算机技术研究所);School of Automation, Southeast University(东南大学自动化学院);College of Science and Engineering, James Cook University(詹姆斯库克大学科学与工程学院);School of Transportation, Southeast University(东南大学交通运输学院)
WaterVG: Waterway Visual Grounding based on Text-Guided Vision and mmWave Radar
Institute of Deep Perception Tech, JITRI(JITRI深度感知技术研究所);School of Advanced Technology, Xi’an Jiaotong-Liverpool University(西交利物浦大学先进技术学院);Faculty of Science and Engineering, University of Liverpool(利物浦大学科学与工程学院);Thrust of Artificial Intelligence and Thrust of Intelligent Transportation, HKUST (Guangzhou)(香港科技大学(广州)人工智能学科及智能交通学科)
Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection
International Digital Economy Academy (IDEA)(国际数字经济学院(IDEA));Institute for AI, Tsinghua-Bosch Joint Center for ML, Tsinghua University(清华大学人工智能研究院、清华-博世联合机器学习中心);The Hong Kong University of Science and Technology(香港科技大学);The Chinese University of Hong Kong (Shenzhen)(香港中文大学(深圳));Microsoft Research(微软研究院);South China University of Technology(华南理工大学)
Grounding Emotion Recognition with Visual Prototypes: VEGA -- Revisiting CLIP in MERC
Xi'an Jiaotong University(西安交通大学);Queen Mary University of London(伦敦女王玛丽大学);Center for Multimodal AI(多模态人工智能中心);Digital Environment Research Institute(数字环境研究院)
GeoWeaver: Grounding Visual Tokens with Geometric Evidence before Scene Reasoning
Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳));Pengcheng Laboratory(鹏城实验室);Pazhou Lab (Huangpu)(琶洲实验室(黄埔));Hainan University(海南大学);University of California at Merced(加州大学默塞德分校)