首页/论文/2003.08717

arXiv 2003.087172021-05-26cs.CVcs.AIcs.CL

Giving Commands to a Self-driving Car: A Multimodal Reasoner for Visual Grounding

KU Leuven（鲁汶大学）

Thierry Deruyttere, Guillem Collell, Marie-Francine Moens

备注：: Updated acknowledgements

阅读 PDF ↗

arXiv 页面 ↗HTML 原文 ↗

英文摘要相关论文

英文摘要：

We propose a new spatial memory module and a spatial reasoner for the Visual Grounding (VG) task. The goal of this task is to find a certain object in an image based on a given textual query. Our work focuses on integrating the regions of a Region Proposal Network (RPN) into a new multi-step reasoning model which we have named a Multimodal Spatial Region Reasoner (MSRR). The introduced model uses the object regions from an RPN as initialization of a 2D spatial memory and then implements a multi-step reasoning process scoring each region according to the query, hence why we call it a multimodal reasoner. We evaluate this new model on challenging datasets and our experiments show that our model that jointly reasons over the object regions of the image and words of the query largely improves accuracy compared to current state-of-the-art models.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

阅读原文

阅读 PDF ↗

arXiv 页面 ↗HTML 原文 ↗

本文内容

英文摘要相关论文

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

继续探索

相关论文

Causal-SAM-LLM：大型语言模型作为因果推理器用于稳健的医学分割

Causal-SAM-LLM: Large Language Models as Causal Reasoners for Robust Medical Segmentation

City University of Hong Kong, Hong Kong, China（香港城市大学）；Xi’an Jiaotong-liverpool University, Suzhou, China（西安交通大学利物浦大学）

Causal-SAM-LLM利用大型语言模型作为因果推理器，提升医学图像分割的鲁棒性和泛化能力。

2026-01-16cs.AIcs.CLcs.CV

Infection-Reasoner: 一种用于伤口感染分类的紧凑视觉-语言模型，结合证据支撑的临床推理

Infection-Reasoner: A Compact Vision-Language Model for Wound Infection Classification with Evidence-Grounded Clinical Reasoning

Worcester Polytechnic Institute（沃斯特理工学院）；UMass Chan Medical School（UMass Chan医学院）

本文提出Infection-Reasoner，一种紧凑的视觉-语言模型，通过两阶段训练提升伤口感染分类和推理生成的准确性与解释性，实验结果显示其在分类和推理质量上优于现有方法。

2026-04-21cs.AIcs.CV

Commands 4 Autonomous Vehicles (C4AV) Workshop Summary

KU Leuven（鲁汶大学）

The task of visual grounding requires locating the most relevant region or object in an image, given a natural language query. So far, progress on this task was mostly measured on curated datasets, which are not always representative of human spoken language. In this work, we deviate from recent, popular task settings and consider the problem under an autonomous vehicle scenario. In particular, we consider a situation where passengers can give free-form natural language commands to a vehicle which can be associated with an object in the street scene. To stimulate research on this topic, we hav…

2020-09-18cs.AIcs.CV

HoneyBee: 用于视觉-语言推理器的数据食谱

HoneyBee: Data Recipes for Vision-Language Reasoners

本文提出HoneyBee数据集，通过数据整理方法提升视觉-语言推理能力，发现上下文来源策略和数据干预显著提升性能，并提出测试时缩放策略以降低解码成本。

2026-03-12cs.CVcs.LG

Predicting Physical World Destinations for Commands Given to Self-Driving Cars

KU Leuven（鲁汶大学）

In recent years, we have seen significant steps taken in the development of self-driving cars. Multiple companies are starting to roll out impressive systems that work in a variety of settings. These systems can sometimes give the impression that full self-driving is just around the corner and that we would soon build cars without even a steering wheel. The increase in the level of autonomy and control given to an AI provides an opportunity for new modes of human-vehicle interaction. However, surveys have shown that giving more control to an AI in self-driving cars is accompanied by a degree o…

2021-12-10cs.AIcs.CLcs.CV

Driving Experience Transfer Method for End-to-End Control of Self-Driving Cars

Electronics and Telecommunications Research Institute（电子通信研究院）

In this paper, we present a transfer learning method for the end-to-end control of self-driving cars, which enables a convolutional neural network (CNN) trained on a source domain to be utilized for the same task in a different target domain. A conventional CNN for the end-to-end control is designed to map a single front-facing camera image to a steering command. To enable the transfer learning, we let the CNN produce not only a steering command but also a lane departure level (LDL) by adding a new task module, which takes the output of the last convolutional layer as input. The CNN trained on…

2018-09-07cs.CVcs.RO

更多相关论文收起更多论文

SciGen-Verifier：科学图像生成中可解释验证的多模态推理器

SciGen-Verifier: A Multimodal Reasoner for Explainable Verification in Scientific Image Generation

South China University of Technology（华南理工大学）；The Hong Kong Polytechnic University（香港理工大学）

本文提出SciGen-Verifier多模态推理验证器及SciGen-Verify基准，通过两阶段强化学习实现科学图像生成的可解释验证与纠错，性能媲美大型专有模型。

2026-09-30cs.CV

End-to-end driving simulation via angle branched network

Sun Yat-sen University（中山大学）；Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）

Imitation learning for end-to-end autonomous driving has drawn attention from academic communities. Current methods either only use images as the input which is ambiguous when a car approaches an intersection, or use additional command information to navigate the vehicle but not automated enough. Focusing on making the vehicle drive along the given path, we propose a new navigation command that does not require human's participation and a novel model architecture called angle branched network. Both the new navigation command and the angle branched network are easy to understand and effective. …

2018-05-19cs.CV

面向端到端自动驾驶的可解释决策

Interpretable Decision-Making for End-to-End Autonomous Driving

Institute for Information Processing, Leibniz University Hannover（信息处理研究所，汉诺威莱布尼茨大学）

针对端到端自动驾驶决策难解释的问题，本文提出用稀疏局部化特征图及相应损失增强控制指令预测的可解释性，并在CARLA上取得最高路线完成率与更低违规分数。

2025-10-21cs.AIcs.CVcs.LG

Learning End-to-end Autonomous Driving using Guided Auxiliary Supervision

Intel Corporation（英特尔公司）

Learning to drive faithfully in highly stochastic urban settings remains an open problem. To that end, we propose a Multi-task Learning from Demonstration (MT-LfD) framework which uses supervised auxiliary task prediction to guide the main task of predicting the driving commands. Our framework involves an end-to-end trainable network for imitating the expert demonstrator's driving commands. The network intermediately predicts visual affordances and action primitives through direct supervision which provide the aforementioned auxiliary supervised guidance. We demonstrate that such joint learnin…

2018-08-30cs.AIcs.CVcs.LG

Ground then Navigate: Language-guided Navigation in Dynamic Scenes

Kohli Center on Intelligent Systems（科利智能系统中心）；International Institute of Information Technology, Hyderabad（海得拉巴国际信息技术学院）

We investigate the Vision-and-Language Navigation (VLN) problem in the context of autonomous driving in outdoor settings. We solve the problem by explicitly grounding the navigable regions corresponding to the textual command. At each timestamp, the model predicts a segmentation mask corresponding to the intermediate or the final navigable region. Our work contrasts with existing efforts in VLN, which pose this task as a node selection problem, given a discrete connected graph corresponding to the environment. We do not assume the availability of such a discretised map. Our work moves towards …

2022-09-24cs.CV

LiDAR-as-Camera for End-to-End Driving

Institute of Computer Science, University of Tartu（塔尔图大学计算机科学研究所）

The core task of any autonomous driving system is to transform sensory inputs into driving commands. In end-to-end driving, this is achieved via a neural network, with one or multiple cameras as the most commonly used input and low-level driving command, e.g. steering angle, as output. However, depth-sensing has been shown in simulation to make the end-to-end driving task easier. On a real car, combining depth and visual information can be challenging, due to the difficulty of obtaining good spatial and temporal alignment of the sensors. To alleviate alignment problems, Ouster LiDARs can outpu…

2022-06-30cs.AIcs.CVcs.RO

Evaluating Robustness over High Level Driving Instruction for Autonomous Driving

ENSTA Paris（巴黎高等先进技术学院）；Institut Polytechnique de Paris（巴黎理工学院）；INRIA（法国国家信息与自动化研究所）；Université Paris-Saclay（巴黎萨克雷大学）；CEA（法国原子能和替代能源委员会）；List（李斯特研究所）

In recent years, we have witnessed increasingly high performance in the field of autonomous end-to-end driving. In particular, more and more research is being done on driving in urban environments, where the car has to follow high level commands to navigate. However, few evaluations are made on the ability of these agents to react in an unexpected situation. Specifically, no evaluations are conducted on the robustness of driving agents in the event of a bad high-level command. We propose here an evaluation method, namely a benchmark that allows to assess the robustness of an agent, and to appr…

2021-05-20cs.AIcs.CVcs.LG

Towards Self-Supervised High Level Sensor Fusion

TUM（慕尼黑工业大学）；Artisense（Artisense公司）；Audi Electronics Venture（奥迪电子创投）

In this paper, we present a framework to control a self-driving car by fusing raw information from RGB images and depth maps. A deep neural network architecture is used for mapping the vision and depth information, respectively, to steering commands. This fusion of information from two sensor sources allows to provide redundancy and fault tolerance in the presence of sensor failures. Even if one of the input sensors fails to produce the correct output, the other functioning sensor would still be able to maneuver the car. Such redundancy is crucial in the critical application of self-driving ca…

2019-02-12cs.AIcs.CVcs.LG