City-VLM：通过多模态不完全学习实现多域感知场景理解
City-VLM: Towards Multidomain Perception Scene Understanding via Multimodal Incomplete Learning
AI总结：
本文提出City-VLM模型和SVM-City多域室外场景数据集，通过不完全多模态学习在缺失模态时构建联合概率分布空间以融合2D与3D信息，在室外场景问答任务上平均超越现有LVLMs 18.14%。
摘要：
场景理解使智能体能够解读并理解其环境。虽然现有用于场景理解的大型视觉-语言模型（LVLMs）主要聚焦于室内家庭任务，但它们在应用于室外大规模场景理解时面临两个显著局限。第一，室外场景通常涵盖更大规模的环境，并通过多种传感器从多个视角（例如鸟瞰视角和地面视角）进行观测，而现有室内LVLMs主要在人形视角下分析建筑尺度环境中的单一视觉模态。第二，现有LVLMs缺乏多域感知室外数据，并且难以有效整合2D与3D视觉信息。为解决上述局限，我们构建了首个多域感知室外场景理解数据集，命名为SVM-City，该数据集来源于多尺度场景，包含多视角和多模态指令微调数据。它包含42万张图像和48.11亿个点云，以及来自车辆、低空无人机、高空航拍飞机和卫星的56.7万对问答数据。为了在缺失一种模态的情况下有效融合多模态数据，我们引入不完全多模态学习来建模室外场景理解，并设计了名为City-VLM的LVLM。多模态融合通过构建联合概率分布空间来实现，而不是直接执行显式融合操作（例如拼接）。在三个典型室外场景理解任务上的实验结果表明，City-VLM在问答任务中平均比现有LVLMs的性能高出18.14%。我们的方法在多个室外场景中展现出实用性和泛化性能。
英文摘要：
Scene understanding enables intelligent agents to interpret and comprehend their environment. While existing large vision-language models (LVLMs) for scene understanding have primarily focused on indoor household tasks, they face two significant limitations when applied to outdoor large-scale scene understanding. First, outdoor scenarios typically encompass larger-scale environments observed through various sensors from multiple viewpoints (e.g., bird view and terrestrial view), while existing indoor LVLMs mainly analyze single visual modalities within building-scale contexts from humanoid viewpoints. Second, existing LVLMs suffer from missing multidomain perception outdoor data and struggle to effectively integrate 2D and 3D visual information. To address the aforementioned limitations, we build the first multidomain perception outdoor scene understanding dataset, named \textbf{\underline{SVM-City}}, deriving from multi\textbf{\underline{S}}cale scenarios with multi\textbf{\underline{V}}iew and multi\textbf{\underline{M}}odal instruction tuning data. It contains $420$k images and $4, 811$M point clouds with $567$k question-answering pairs from vehicles, low-altitude drones, high-altitude aerial planes, and satellite. To effectively fuse the multimodal data in the absence of one modality, we introduce incomplete multimodal learning to model outdoor scene understanding and design the LVLM named \textbf{\underline{City-VLM}}. Multimodal fusion is realized by constructing a joint probabilistic distribution space rather than implementing directly explicit fusion operations (e.g., concatenation). Experimental results on three typical outdoor scene understanding tasks show City-VLM achieves $18.14 \%$ performance surpassing existing LVLMs in question-answering tasks averagely. Our method demonstrates pragmatic and generalization performance across multiple outdoor scenes.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
