DeepSight: 通过深度驱动的多模态模型连接深度图与语言
DeepSight: Bridging Depth Maps and Language with a Depth-Driven Multimodal Model
AI总结：
DeepSight通过深度驱动的多模态模型提升三维场景理解，利用深度图像特性增强空间推理能力，并通过新数据集和模型改进实现了深度感知和任务性能的显著提升。
摘要：
多模态大语言模型（MLLMs）在图像描述和视觉问答（VQA）等任务中取得了显著成效；然而，它们往往难以准确解释视觉数据中固有的深度信息。在本工作中，我们引入DeepSight，首个专门针对深度的MLLM，旨在提升三维场景理解。不同于传统方法将RGB图像编码与文本对齐，我们的方法利用深度图像的独特特性：单通道灰度图像，其中像素值直接反映深度线索，以提升空间推理能力。为解决深度数据有限和简单通道复制不足的问题，我们构建了新的深度图像-文本配对数据集和深度指令数据集。深度图通过GLPN模型从视觉图像生成，GPT-4用于整理对应的深度指令，该方法已通过LLaVA验证。此外，我们修改CLIP中的ViT编码器以纳入局部物体信息，从而更有效地捕捉深度的细微连续变化。为评估模型性能，我们基于现有深度图像数据集开发了全面的深度问答基准，严格评估典型深度图场景的理解能力。实验结果表明，DeepSight显著提升了深度感知和下游任务性能，标志着多模态三维理解迈出了重要一步。
英文摘要：
Multimodal large language models (MLLMs) have achieved impressive performance across various tasks such as image captioning and visual question answer(VQA); however, they often struggle to accurately interpret depth information inherent in visual data. In this work, we introduce DeepSight, the first dedicated depth MLLM designed to enhance three-dimensional scene understanding. Unlike conventional methods that align RGB image encodings with text, our approach takes advantage of the unique characteristics of depth images: single-channel grayscale images where the pixel values directly reflect depth cues to improve spatial reasoning. To address challenges associated with limited depth data and the inadequacy of simple channel replication, we construct a novel depth image-text pair dataset and a depth instruction dataset. Depth maps are generated from visual images using the GLPN model, and GPT-4 is employed to curate corresponding depth instructions, an approach validated by LLaVA. Additionally, we modify the ViT encoder in CLIP to incorporate local object information, thereby capturing the subtle continuous variations of depth more effectively. To evaluate the performance of our model, we develop a comprehensive depth question answer benchmark based on existing depth image datasets, which rigorously assesses understanding in typical depth map scenarios. Experimental results demonstrate that DeepSight significantly enhances depth perception and downstream task performance, marking a substantial step forward in multimodal three-dimensional understanding.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
