GPT-4增强的多模态接地用于自动驾驶：利用跨模态注意力与大语言模型
GPT-4 Enhanced Multimodal Grounding for Autonomous Driving: Leveraging Cross-Modal Attention with Large Language Models
AI总结：
本文提出了一种结合GPT-4的大语言模型的多模态接地框架，用于提升自动驾驶中的视觉理解和指令执行能力。
摘要：
在自动驾驶（AV）领域，准确区分指挥意图并在视觉上下文中执行语言指令是一项重大挑战。本文介绍了一个复杂的编码器-解码器框架，旨在解决AV中的视觉接地问题。我们的上下文感知视觉接地（CAVG）模型是一个先进的系统，集成了五个核心编码器-文本、图像、上下文、跨模态-以及一个多模态解码器。这种集成使CAVG模型能够有效地捕捉上下文语义，并通过最先进的大语言模型（LLMs）包括GPT-4来学习人类情感特征。CAVG的架构通过实现多头跨模态注意力机制和区域特定动态（RSD）层进行加强，用于注意力调节。这种架构设计使模型能够高效地处理和解释多种跨模态输入，从而全面理解语言指令与相应视觉场景之间的相关性。在Talk2Car数据集上的实证评估表明，CAVG在预测准确性和操作效率方面建立了新的标准。值得注意的是，即使在训练数据有限的情况下，模型的表现也十分突出，训练数据量从完整数据集的50%到75%不等。这一特性突显了其有效性和在实际AV应用中的潜力。此外，CAVG在具有挑战性的场景中表现出显著的鲁棒性和适应性，包括长文本指令解释、低光条件、模糊指令上下文、恶劣天气条件以及人口密集的城市环境。所提出模型的代码可在我们的GitHub上获得。
英文摘要：
In the field of autonomous vehicles (AVs), accurately discerning commander intent and executing linguistic commands within a visual context presents a significant challenge. This paper introduces a sophisticated encoder-decoder framework, developed to address visual grounding in AVs.Our Context-Aware Visual Grounding (CAVG) model is an advanced system that integrates five core encoders-Text, Image, Context, and Cross-Modal-with a Multimodal decoder. This integration enables the CAVG model to adeptly capture contextual semantics and to learn human emotional features, augmented by state-of-the-art Large Language Models (LLMs) including GPT-4. The architecture of CAVG is reinforced by the implementation of multi-head cross-modal attention mechanisms and a Region-Specific Dynamic (RSD) layer for attention modulation. This architectural design enables the model to efficiently process and interpret a range of cross-modal inputs, yielding a comprehensive understanding of the correlation between verbal commands and corresponding visual scenes. Empirical evaluations on the Talk2Car dataset, a real-world benchmark, demonstrate that CAVG establishes new standards in prediction accuracy and operational efficiency. Notably, the model exhibits exceptional performance even with limited training data, ranging from 50% to 75% of the full dataset. This feature highlights its effectiveness and potential for deployment in practical AV applications. Moreover, CAVG has shown remarkable robustness and adaptability in challenging scenarios, including long-text command interpretation, low-light conditions, ambiguous command contexts, inclement weather conditions, and densely populated urban environments. The code for the proposed model is available at our Github.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
