EgoDTM: 向3D感知的自体视频-语言预训练迈进
EgoDTM: Towards 3D-Aware Egocentric Video-Language Pretraining
AI总结：
EgoDTM通过结合大规模3D感知视频预训练和视频-文本对比学习，提升视频-语言模型的3D感知能力，实现更丰富的空间理解。
摘要：
自体视频-语言预训练显著推动了视频表示学习的发展。人类感知和与一个完全三维世界互动，发展出超越基于文本理解的空间意识。然而，大多数先前的工作从1D文本或2D视觉线索（如边界框）中学习，这些本质上缺乏三维理解。为了弥合这一差距，我们引入EgoDTM，一种自体深度和文本感知模型，通过大规模3D感知视频预训练和视频-文本对比学习共同训练。EgoDTM集成了一个轻量级的3D感知解码器，以高效地从由深度估计模型生成的伪深度图中学习3D感知。为进一步促进3D感知视频预训练，我们通过有机结合几个基础模型，丰富原始简短的描述，加入手-物体视觉线索。广泛实验表明，EgoDTM在各种下游任务中均表现出优越的性能，突显了其优越的3D感知视觉理解。代码：https://github.com/xuboshen/EgoDTM。
英文摘要：
Egocentric video-language pretraining has significantly advanced video representation learning. Humans perceive and interact with a fully 3D world, developing spatial awareness that extends beyond text-based understanding. However, most previous works learn from 1D text or 2D visual cues, such as bounding boxes, which inherently lack 3D understanding. To bridge this gap, we introduce EgoDTM, an Egocentric Depth- and Text-aware Model, jointly trained through large-scale 3D-aware video pretraining and video-text contrastive learning. EgoDTM incorporates a lightweight 3D-aware decoder to efficiently learn 3D-awareness from pseudo depth maps generated by depth estimation models. To further facilitate 3D-aware video pretraining, we enrich the original brief captions with hand-object visual cues by organically combining several foundation models. Extensive experiments demonstrate EgoDTM's superior performance across diverse downstream tasks, highlighting its superior 3D-aware visual understanding. Code: https://github.com/xuboshen/EgoDTM.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
