ReLift:作为提升轨迹修复的密集3D跟踪
ReLift: Dense 3D Tracking as Lifted Trajectory Repair
AI总结:
ReLift将密集3D跟踪转化为几何轨迹修复,利用预训练2D跟踪器与3D几何模型,在WorldTrack上实现最先进精度,且可复用、运行高效。
摘要:
从单目视频进行密集3D跟踪需要建立随时间变化的点对应关系、估计其深度,并在遮挡期间推断其轨迹。现有方法通常联合学习这些任务,考虑到4D监督的稀缺性,这是一项要求很高的目标。我们表明,强大的预训练2D跟踪器和3D几何模型提供了有效的基础:提升轨迹(其中被遮挡段仅通过插值得到)已经可与使用相同几何结构的专用3D跟踪器相媲美。然而,提升轨迹继承了对应关系和深度误差,在遮挡情况下,其深度来自遮挡物而非被跟踪点。我们引入ReLift,这是一个高效的前馈框架,将密集3D跟踪重新表述为几何轨迹修复与补全。其核心是一个轨迹修正器,沿轨迹和跨轨迹交替进行注意力操作,以保留可靠观测值、修复损坏位置并补全缺失运动。仅使用几何输入使我们能够直接在带有模拟提升误差的3D轨迹上进行训练,将训练与图像以及任何特定跟踪器或几何模型解耦。将4D监督集中在轨迹细化上,可得到一个可复用的模型,该模型无需重新训练即可持续改进不同跟踪器和几何提供者的提升轨迹,因此可直接受益于任一组件的未来进展。在WorldTrack数据集上,ReLift实现了最先进的平均精度,在所有点上比最强基线提升了2.1 APD,在动态点上提升了3.0 APD,其更快的配置仍可与领先的3D跟踪器竞争,同时运行速度比4RC更快。访问我们的补充页面可获取更多示例和代码。
英文摘要:
Dense 3D tracking from monocular video requires establishing point correspondences over time, estimating their depth, and inferring their trajectories during occlusion. Existing methods typically learn these tasks jointly, a demanding objective given the scarcity of 4D supervision. We show that strong pretrained 2D trackers and 3D geometry models provide an effective foundation: lifted trajectories, with occluded segments simply interpolated, already rival dedicated 3D trackers using the same geometry. However, lifted trajectories inherit correspondence and depth errors, and under occlusion, their depth comes from the occluder rather than the tracked point. We introduce ReLift, an efficient feed-forward framework that recasts dense 3D tracking as geometric trajectory repair and completion. At its core, a trajectory corrector alternates attention along and across trajectories to preserve reliable observations, repair corrupted positions, and complete missing motion. Using only geometric inputs allows us to train directly on 3D trajectories with simulated lifting errors, decoupling training from images and from any specific tracker or geometry model. Concentrating 4D supervision on trajectory refinement yields a reusable model that consistently improves lifted trajectories across trackers and geometry providers without retraining, so it benefits directly from future advances in either component. On WorldTrack, ReLift achieves state-of-the-art mean accuracy, improving on the strongest baselines by 2.1 APD on all points and 3.0 on dynamic points, with faster configurations remaining competitive with leading 3D trackers while running faster than 4RC. Visit our supplementry page for more examples and code.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。