Info3R：面向3D重建的信息自适应测试时训练

中文摘要 AI 辅助

基于Transformer的模型近期在从图像进行3D重建方面取得了强劲性能，近期工作将其扩展为以在线方式处理视频流，以用于现实世界部署。然而，现有方法在处理长图像流时忽略两个关键信号：每个输入帧的重要性以及模型内部状态的信息饱和。本文提出Info3R，一种新颖的信息自适应测试时训练方法，用于在线3D重建。我们引入信息感知的状态更新，根据每个输入帧的冗余度和信息量来调节状态更新的强度。为恢复状态的可塑性——即其纳入新观测的能力——我们提出动态状态重置，由状态更新的累积幅度和模型的预测置信度触发，并伴随锚点到世界的对齐。我们的方法在相机位姿估计、视频深度估计和3D重建上取得一致的改进，同时大幅缓解长序列评估中的性能退化。值得注意的是，在KITTI Odometry上，我们的方法平均达到比LongStream低1.68倍的ATE，展示了其在扩展户外序列上的鲁棒性。

英文摘要

Transformer-based models have recently achieved strong performance on 3D reconstruction from images, and recent works extend them to process video streams in an online manner for real-world deployment. However, existing methods overlook two key signals when handling long image streams: the importance of each incoming frame and the information saturation of the model's internal state. In this paper, we propose Info3R, a novel information-adaptive test-time training method for the online 3D reconstruction. We introduce an information-aware state update that modulates the state update strength based on the redundancy and informativeness of each incoming frame. To restore the state's plasticity -- its capacity to incorporate new observations -- we propose a dynamic state reset, triggered by the cumulative magnitude of state updates and the model's prediction confidence and accompanied by an anchor-to-world alignment. Our method achieves consistent improvements on camera pose estimation, video depth estimation, and 3D reconstruction, while substantially mitigating the performance degradation in the long sequence evaluation. Notably, on KITTI Odometry, our method achieves on average 1.68x lower ATE than LongStream, demonstrating its robustness on extended outdoor sequences.

发表机构

• Korea Advanced Institute of Science and Technology（韩国科学技术院）

机构由 AI 辅助整理，请以论文原文为准。