We present a unified framework capable of solving a broad range of 3D tasks. Our approach features a stateful recurrent model that continuously updates its state representation with each new observation. Given a stream of images, this evolving state can be used to generate metric-scale pointmaps (per-pixel 3D points) for each new input in an online fashion. These pointmaps reside within a common coordinate system, and can be accumulated into a coherent, dense scene reconstruction that updates as new images arrive. Our model, called CUT3R (Continuous Updating Transformer for 3D Reconstruction), captures rich priors of real-world scenes: not only can it predict accurate pointmaps from image observations, but it can also infer unseen regions of the scene by probing at virtual, unobserved views. Our method is simple yet highly flexible, naturally accepting varying lengths of images that may be either video streams or unordered photo collections, containing both static and dynamic content. We evaluate our method on various 3D/4D tasks and demonstrate competitive or state-of-the-art performance in each. Project Page: https://cut3r.github.io/
Persistent Visual Memory: Sustaining Perception for Deep Generation in LVLMs
Shanghai AI Laboratory(上海人工智能实验室);Shanghai Jiao Tong University(上海交通大学);The Chinese University of Hong Kong(香港中文大学);Nanjing University(南京大学);Tongji University(同济大学);Wuhan University(武汉大学)
Belief Consistency Between Foundation-Model Evidence and Geometric Perception in Persistent Robotic Maps
Department of Computer Science, University of Colorado, Boulder(科罗拉多大学博尔德分校计算机科学系);Computer Science and Artificial Intelligence Lab, Massachusetts Institute of Technology(麻省理工学院计算机科学与人工智能实验室)
CoSpace: Benchmarking Continuous Space Perception Ability for Vision-Language Models
Tsinghua University(清华大学);Institute for AI Industry Research (AIR), Tsinghua University(清华大学人工智能产业研究院);University of Science and Technology Beijing(北京科技大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室)
Continuous Perception for Classifying Shapes and Weights of Garmentsfor Robotic Vision Applications
University of Glasgow(格拉斯哥大学)
我们提出了一种方法来连续感知机器人洗衣任务。我们的假设是,服装的形状和重量的视觉预测是可能的,通过神经网络,从视频序列中学习服装的动态变化。在训练过程中,通过输入连续的帧来利用连续的感知,其中网络学习服装如何变形。为了评估我们的假设,我们在操纵服装时捕获了40 K RGB和40 K深度视频序列的数据集。我们还进行了消融研究,以了解神经网络是否学习服装的物理和动态特性。我们的研究结果表明,修改后的AlexNet-LSTM架构对服装的形状和重量具有最佳的分类性能。为了进一步提供证据证明连续感知有助于预测服装的形状和重量,我们在看不见的视频序列上评估了我们的网络,并计算了一系列预测的“移动平均值”。我们发现,我们的网络对服装的形状和重量的分类准确率分别为48%和60%。
MooD: Perception-Enhanced Efficient Affective Image Editing via Continuous Valence-Arousal Modeling
School of Cyber Science and Engineering, Zhengzhou University(郑州大学信息科学与工程学院);School of Computer Science and Technology, Xi’an Jiaotong University(西安交通大学计算机科学与技术学院);School of Journalism and New Media, Xi’an Jiaotong University(西安交通大学新闻与传播学院);Research Center for Space Computing System, Zhejiang Lab(浙江实验室空间计算系统研究中心)
Embodied VideoAgent: Persistent Memory from Egocentric Videos and Embodied Sensors Enables Dynamic Scene Understanding
State Key Laboratory of General Artificial Intelligence, BIGAI(通用人工智能国家重点实验室);University of Science and Technology of China(中国科学技术大学);Tsinghua University(清华大学);Peking University(北京大学)
ST-Booster: An Iterative SpatioTemporal Perception Booster for Vision-and-Language Navigation in Continuous Environments
Robotics and Control Laboratory, the School of Advanced Manufacturing and Robotics, and the State Key Laboratory of Turbulence and Complex Systems, Peking University(机器人与控制实验室、先进制造与机器人学院、湍流与复杂系统国家重点实验室,北京大学);Defense Innovation Institute, Academy of Military Sciences(国防创新研究院,军事科学院);Tianjin Artificial Intelligence Innovation Center(天津人工智能创新中心);Harbin Institute of Technology(哈尔滨工业大学)