We consider the problem of efficiently representing casually captured monocular videos in a spatially- and temporally-coherent manner. While existing approaches predominantly rely on 2D/2.5D techniques treating videos as collections of spatiotemporal pixels, they struggle with complex motions, occlusions, and geometric consistency due to absence of temporal coherence and explicit 3D structure. Drawing inspiration from monocular video as a projection of the dynamic 3D world, we explore representing videos in their intrinsic 3D form through continuous flows of Gaussian primitives in space-time. In this paper, we propose NutWorld, a novel framework that efficiently transforms monocular videos into dynamic 3D Gaussian representations in a single forward pass. At its core, NutWorld introduces a structured spatial-temporal aligned Gaussian (STAG) representation, enabling optimization-free scene modeling with effective depth and flow regularization. Through comprehensive experiments, we demonstrate that NutWorld achieves high-fidelity video reconstruction quality while enabling various downstream applications in real-time. Demos and code will be available at https://github.com/Nut-World/NutWorld.
In this paper, we study the problem of reproducing the world lighting from a single image of an object covered with random specular microfacets on the surface. We show that such reflectors can be interpreted as a randomized mapping from the lighting to the image. Such specular objects have very different optical properties from both diffuse surfaces and smooth specular objects like metals, so we design special imaging system to robustly and effectively photograph them. We present simple yet reliable algorithms to calibrate the proposed system and do the inference. We conduct experiments to ver…
We address the dual problems of novel view synthesis and environment reconstruction from hand-held RGBD sensors. Our contributions include 1) modeling highly specular objects, 2) modeling inter-reflections and Fresnel effects, and 3) enabling surface light field reconstruction with the same input needed to reconstruct shape alone. In cases where scene surface has a strong mirror-like material component, we generate highly detailed environment images, revealing room composition, objects, people, buildings, and trees visible through windows. Our approach yields state of the art view synthesis te…
The All-Seeing Project: Towards Panoptic Visual Recognition and Understanding of the Open World
Fudan University（复旦大学）；Huazhong University of Science and Technology（华中科技大学）；Harbin Institute of Technology（哈尔滨工业大学）；Shanghai AI Laboratory（上海人工智能实验室）；Nanjing University（南京大学）；The Chinese University of Hong Kong（香港中文大学）；SenseTime Research（商汤科技研究院）；Tsinghua University（清华大学）
本文提出 The All-Seeing 项目，构建了包含超 10 亿区域标注的 AS-1B 数据集，并开发了统一框架 ASM，通过开放式语言提示和位置训练，在多种视觉语言任务中实现卓越的零样本性能。
In this paper we introduce the TorontoCity benchmark, which covers the full greater Toronto area (GTA) with 712.5 $km^2$ of land, 8439 $km$ of road and around 400,000 buildings. Our benchmark provides different perspectives of the world captured from airplanes, drones and cars driving around the city. Manually labeling such a large scale dataset is infeasible. Instead, we propose to utilize different sources of high-precision maps to create our ground truth. Towards this goal, we develop algorithms that allow us to align all data sources with the maps while requiring minimal human supervision.…
The All-Seeing Project V2: Towards General Relation Comprehension of the Open World
OpenGVLab（开放视觉实验室）；Shanghai AI Laboratory（上海人工智能实验室）；Fudan University（复旦大学）；Tsinghua University（清华大学）；Nanjing University（南京大学）；The Chinese University of Hong Kong（香港中文大学）；Harbin Institute of Technology（哈尔滨工业大学）；SenseTime Research（商汤科技研究院）
Hefei University of Technology（合肥工业大学）；Zhengzhou University（郑州大学）；Zhejiang University（浙江大学）；Sea AI Lab（Sea人工智能实验室）
Low-light image enhancement (LLIE) aims at improving the illumination and visibility of dark images with lighting noise. To handle the real-world low-light images often with heavy and complex noise, some efforts have been made for joint LLIE and denoising, which however only achieve inferior restoration performance. We attribute it to two challenges: 1) in real-world low-light images, noise is somewhat covered by low-lighting and the left noise after denoising would be inevitably amplified during enhancement; 2) conversion of raw data to sRGB would cause information loss and also more noise, a…
Istituto Italiano di Tecnologia（意大利技术研究院）；Università degli Studi di Genova（热那亚大学）；Huawei Technologies Ltd., Ireland Research Center（华为技术有限公司爱尔兰研究中心）；University of Verona（维罗纳大学）
In Generalized Zero-Shot Learning (GZSL), unseen categories (for which no visual data are available at training time) can be predicted by leveraging their class embeddings (e.g., a list of attributes describing them) together with a complementary pool of seen classes (paired with both visual data and class embeddings). Despite GZSL is arguably challenging, we posit that knowing in advance the class embeddings, especially for unseen categories, is an actual limit of the applicability of GZSL towards real-world scenarios. To relax this assumption, we propose Open Zero-Shot Learning (OZSL) to ext…
From Seeing to Predicting: A Vision-Language Framework for Trajectory Forecasting and Controlled Video Generation
Foundation Model Research Center, Institute of Automation, Chinese Academy of Sciences（基础模型研究中心、自动化研究所、中国科学院）；Peng Cheng Laboratory, Shenzhen, China（鹏城实验室、深圳中国）；School of Artificial Intelligence, University of Chinese Academy of Science, Beijing, China（人工智能学院、中国科学院大学、北京中国）；Wuhan AI Research, Wuhan, China（武汉人工智能研究、武汉中国）；MAPLE Lab, Westlake University（MAPLE实验室、西湖大学）
Seeing Through Uncertainty: Free-Energy-Inspired Real-Time Adaptation for Robust Visual Navigation
Department of Computer Science, The University of Manchester（曼彻斯特大学计算机科学系）；Institute for Infocomm Research (I2R), Agency for Science, Technology and Research (A*STAR)（信息与通信研究院（I2R）、科技研究局（A*STAR））；College of Computing and Data Science, Nanyang Technological University（南洋理工大学计算与数据科学学院）
Can We Talk Models Into Seeing the World Differently?
Offenburg University（奥芬堡大学）；University of Mannheim（曼海姆大学）；Tübingen AI Center（蒂宾根人工智能中心）；Goethe University Frankfurt（法兰克福大学）；University of Siegen（锡根大学）；Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）；Google DeepMind（谷歌DeepMind）；MIT CSAIL（麻省理工学院计算机科学与人工智能实验室）
Institute for Systems and Robotics（系统与机器人研究所）；Université de Toulon（土伦大学）；Aix Marseille University（阿维尼翁-马赛大学）；CNRS（国家科学研究中心）；LIS Toulon, France（土伦法国实验室）
University of Barcelona（巴塞罗那大学）；Telefonica I+D, Research（西班牙电信研发研究部）；Institut de Robòtica i Informàtica Industrial (CSIC-UPC)（机器人与工业信息研究所（CSIC-UPC））
Our interaction with the world is an inherently multimodal experience. However, the understanding of human-to-object interactions has historically been addressed focusing on a single modality. In particular, a limited number of works have considered to integrate the visual and audio modalities for this purpose. In this work, we propose a multimodal approach for egocentric action recognition in a kitchen environment that relies on audio and visual information. Our model combines a sparse temporal sampling strategy with a late fusion of audio, spatial, and temporal streams. Experimental results …
Seeing Together: Multi-Robot Cooperative Egocentric Spatial Reasoning with Multimodal Large Language Models
Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）；Hunan University（湖南大学）；University of Oxford（牛津大学）；Zhejiang University（浙江大学）；ETH Zurich（苏黎世联邦理工学院）；Ant Group（蚂蚁集团）
Seeing is Not Reasoning: MVPBench for Graph-based Evaluation of Multi-path Visual Physical CoT
Central South University（中南大学）；University of Electronic Science and Technology of China（电子科技大学）；Peng Cheng Laboratory（鹏城实验室）；Nanjing University（南京大学）；Microsoft（微软公司）
Centre for Intelligent Systems（智能系统中心）；Technion, Israel Institute of Technology（以色列理工学院）
Humans possess an intricate and powerful visual system in order to perceive and understand the environing world. Human perception can effortlessly detect and correctly group features in visual data and can even interpret random-dot videos induced by imaging natural dynamic scenes with highly noisy sensors such as ultrasound imaging. Remarkably, this happens even if perception completely fails when the same information is presented frame by frame rather than in a video sequence. We study this property of surprising dynamic perception with the first goal of proposing a new detection and spatio-t…
Seeing through Light and Darkness: Sensor-Physics Grounded Deblurring HDR NeRF from Single-Exposure Images and Events
State Key Laboratory of Virtual Reality Technology and Systems, SCSE & QRI, Beihang University（北京航空航天大学虚拟现实技术与系统国家重点实验室）；School of Artificial Intelligence, Beijing Normal University（北京师范大学人工智能学院）
Seeing Through Fog: Towards Fog-Invariant Action Recognition
Beijing Institute of Technology, Beijing, China（北京理工大学，北京，中国）；Beijing Institute for General Artificial Intelligence, Beijing, China（北京通用人工智能研究院，北京，中国）；Yangtze Delta Region Academy of Beijing Institute of Technology, Jiaxing, China（北京理工大学扬子江地区研究院，嘉兴，中国）
Seeing the Forest and the Trees: Query-Aware Tokenizer for Long-Video Multimodal Language Models
Queen Mary University of London（伦敦女王学院）；University of Sheffield（谢菲尔德大学）；Imperial College London（伦敦帝国学院）；Pengcheng Laboratory（鹏城实验室）；Meta Inc（Meta公司）；Meituan Inc（美团公司）；Nanjing University of Science（南京理工大学）；University of Birmingham（伯明翰大学）；Utrecht University（乌得勒支大学）
