EgoTrack3D：一种用于第一人称视角三维目标跟踪的模块化框架

中文摘要 AI 辅助

从第一人称视角视频理解三维场景是机器人学与自主导航的基础，但快速的视角变化和部分遮挡使得构建结构化表示极具挑战性。现有的三维跟踪及场景图构建方法主要针对显式交互或假设静态场景，限制了其捕捉复杂动态的能力。本文提出EgoTrack3D，一种可直接从第一人称视角RGB视频重建并维护动态三维场景表示的模块化框架。该框架将二维分割掩码提升至全局三维坐标系，采用基于点的运动评分机制与基于体素的合并启发式方法关联目标轨迹。EgoTrack3D可随时间维持准确的表示，在Aria Digital Twin（ADT）数据集上，其正确位置百分比（PCL）较最强基线提升11%，同时解决了静态与动态目标的持久三维跟踪这一更通用的设置。此外，为验证系统在模拟真实部署约束的退化条件下的鲁棒性，我们用稀疏三维边界框估计替换密集深度图，并集成交互引导的动态关联，使EgoTrack3D即便在观测含噪的情况下仍能维持准确的空间表示。

英文摘要

Understanding 3D scenes from egocentric video is fundamental for robotics and autonomous navigation, yet rapid viewpoint changes and partial occlusions make building structured representations challenging. Existing 3D tracking and scene graph construction methods primarily address explicit interactions or assume static scenes, limiting their ability to capture complex dynamics. We introduce EgoTrack3D, a modular framework that reconstructs and maintains a dynamic 3D scene representation directly from egocentric RGB video. The framework lifts 2D segmentation masks into a global 3D coordinate frame, using a point-based motion scoring mechanism alongside a voxel-based merging heuristic to associate object tracks. EgoTrack3D maintains accurate representations over time, achieving an 11% improvement in percentage of correct locations (PCL) relative to the strongest baseline on the Aria Digital Twin (ADT) dataset, while addressing the more general setting of persistent 3D tracking for both static and dynamic objects. Furthermore, to demonstrate the system's robustness under degraded conditions that simulate real-world deployment constraints, we replace dense depth maps with sparse 3D bounding box estimation and integrate interaction-guided dynamic association, enabling EgoTrack3D to maintain accurate spatial representations despite noisy observations.

发表机构

• ETH Zürich（苏黎世联邦理工学院）
• Microsoft Research（微软研究院）
• TUM（慕尼黑工业大学）
• MCML（慕尼黑计算与机器学习中心）

机构由 AI 辅助整理，请以论文原文为准。