TrackCraft3R：将视频扩散变换器重新用于密集3D跟踪
TrackCraft3R: Repurposing Video Diffusion Transformers for Dense 3D Tracking
AI总结：
本文提出TrackCraft3R，通过重新利用预训练的视频扩散变换器，实现了基于单目视频的密集3D跟踪，采用双潜表示和时间RoPE对齐设计，提升了跟踪性能和效率。
摘要：
从单目视频中进行密集3D跟踪是动态场景理解的基础。尽管最近的3D基础模型提供了可靠的每帧几何信息，但在此几何中恢复物体运动仍然具有挑战性，需要从真实世界视频中学习强运动先验。现有的3D跟踪器要么遵循迭代范式，从合成数据中训练，要么微调从静态多视图图像中学习的3D重建模型，两者都缺乏真实世界的运动先验。预训练的视频扩散变换器（视频DiTs）提供了丰富的时空先验，来自互联网级别的视频，使其成为3D跟踪的有希望的基础。然而，其帧锚定的公式，生成每个帧的内容，本质上与参考锚定的密集3D跟踪不匹配，后者必须在时间上遵循参考帧的相同物理点。我们提出了TrackCraft3R，这是首个将视频DiT重新用于馈送密集3D跟踪的方法。给定一个单目视频及其帧锚定的重建点图，TrackCraft3R预测一个参考锚定的跟踪点图，该点图在时间上遵循第一帧的每个像素，并附带其可见性。我们通过两种设计实现这一点：（i）双潜表示，使用每帧几何潜码和参考锚定跟踪潜码作为密集查询；（ii）时间RoPE对齐，指定每个跟踪潜码的目标时间戳。共同，这些设计将视频DiT的每帧生成范式转换为参考锚定的跟踪范式，通过LoRA微调。TrackCraft3R在标准稀疏和密集3D跟踪基准上实现了最先进的性能，同时比最强的前方法快1.3倍，使用峰值内存少4.6倍。我们进一步展示了对大运动和长视频的鲁棒性。
英文摘要：
Dense 3D tracking from monocular video is fundamental to dynamic scene understanding. While recent 3D foundation models provide reliable per-frame geometry, recovering object motion in this geometry remains challenging and benefits from strong motion priors learned from real-world videos. Existing 3D trackers either follow iterative paradigms trained from scratch on synthetic data or fine-tune 3D reconstruction models learned from static multi-view images, both lacking real-world motion priors. Pre-trained video diffusion transformers (video DiTs) offer rich spatio-temporal priors from internet-scale videos, making them a promising foundation for 3D tracking. However, their frame-anchored formulation, which generates each frame's content, is fundamentally mismatched with reference-anchored dense 3D tracking, which must follow the same physical points from a reference frame across time. We present TrackCraft3R, the first method to repurpose a video DiT as a feed-forward dense 3D tracker. Given a monocular video and its frame-anchored reconstruction pointmap, TrackCraft3R predicts a reference-anchored tracking pointmap that follows every pixel of the first frame across time in a single forward pass, along with its visibility. We achieve this through two designs: (i) a dual-latent representation that uses per-frame geometry latents and reference-anchored track latents as dense queries, and (ii) temporal RoPE alignment, which specifies the target timestamp of each track latent. Together, these designs convert the per-frame generative paradigm of video DiTs into a reference-anchored tracking formulation with LoRA fine-tuning. TrackCraft3R achieves state-of-the-art performance on standard sparse and dense 3D tracking benchmarks, while running 1.3x faster and using 4.6x less peak memory than the strongest prior method. We further demonstrate robustness to large motions and long videos.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
