In advanced mission concepts with high levels of autonomy, spacecraft need to internally model the pose and shape of nearby orbiting objects. Recent works in neural scene representations show promising results for inferring generic three-dimensional scenes from optical images. Neural Radiance Fields (NeRF) have shown success in rendering highly specular surfaces using a large number of images and their pose. More recently, Generative Radiance Fields (GRAF) achieved full volumetric reconstruction of a scene from unposed images only, thanks to the use of an adversarial framework to train a NeRF. In this paper, we compare and evaluate the potential of NeRF and GRAF to render novel views and extract the 3D shape of two different spacecraft, the Soil Moisture and Ocean Salinity satellite of ESA's Living Planet Programme and a generic cube sat. Considering the best performances of both models, we observe that NeRF has the ability to render more accurate images regarding the material specularity of the spacecraft and its pose. For its part, GRAF generates precise novel views with accurate details even when parts of the satellites are shadowed while having the significant advantage of not needing any information about the relative pose.
University of California, Santa Barbara（加州大学圣巴巴拉分校）；University of California（加州大学）
Retinal degenerative diseases cause profound visual impairment in more than 10 million people worldwide, and retinal prostheses are being developed to restore vision to these individuals. Analogous to cochlear implants, these devices electrically stimulate surviving retinal cells to evoke visual percepts (phosphenes). However, the quality of current prosthetic vision is still rudimentary. Rather than aiming to restore "natural" vision, there is potential merit in borrowing state-of-the-art computer vision algorithms as image processing techniques to maximize the usefulness of prosthetic vision…
SceneSplat: Gaussian Splatting-based Scene Understanding with Vision-Language Pretraining
University of Amsterdam（阿姆斯特丹大学）；Computer Vision Lab, ETH Zurich（苏黎世联邦理工学院计算机视觉实验室）；INSAIT, Sofia University ”St. Kliment Ohridski”（索菲亚大学”圣克莱门特·欧赫里迪斯”研究所）；Nanjing University of Aeronautics and Astronautics（南京航空航天大学）；University of Pisa（比萨大学）；University of Trento（特伦特大学）
3D Smoke Scene Reconstruction Guided by Vision Priors from Multimodal Large Language Models
Hefei University of Technology（合肥工业大学）；Institute of Artificial Intelligence, Hefei Comprehensive National Science Center（合肥综合性国家科学中心人工智能研究院）；Anhui University（安徽大学）；United Arab Emirates University（阿联酋大学）；Nanyang Technological University（南洋理工大学）
SceneBench: A Hierarchical Benchmark for Vision-Language Understanding of 3D Scenes
Nepal Applied Mathematics and Informatics Institute for research (NAAMII)（尼泊尔应用数学与信息学研究所）；INSAIT, Sofia University（索非亚大学INSAIT）；State University of New York at Stony Brook（纽约州立大学石溪分校）
We present a system for applying sim2real approaches to "in the wild" scenes with realistic visuals, and to policies which rely on active perception using RGB cameras. Given a short video of a static scene collected using a generic phone, we learn the scene's contact geometry and a function for novel view synthesis using a Neural Radiance Field (NeRF). We augment the NeRF rendering of the static scene by overlaying the rendering of other dynamic objects (e.g. the robot's own body, a ball). A simulation is then created using the rendering engine in a physics simulator which computes contact dyn…
On-board estimation of the pose of an uncooperative target spacecraft is an essential task for future on-orbit servicing and close-proximity formation flying missions. However, two issues hinder reliable on-board monocular vision based pose estimation: robustness to illumination conditions due to a lack of reliable visual features and scarcity of image datasets required for training and benchmarking. To address these two issues, this work details the design and validation of a monocular vision based pose determination architecture for spaceborne applications. The primary contribution to the st…
Video scene parsing in the wild with diverse scenarios is a challenging and great significance task, especially with the rapid development of automatic driving technique. The dataset Video Scene Parsing in the Wild(VSPW) contains well-trimmed long-temporal, dense annotation and high resolution clips. Based on VSPW, we design a Temporal Bilateral Network with Vision Transformer. We first design a spatial path with convolutions to generate low level features which can preserve the spatial information. Meanwhile, a context path with vision transformer is employed to obtain sufficient context info…
Nitte (Deemed to Be University), Nitte Institute of Professional Education（尼特大学尼特专业教育学院）；Ballari Institute of Technology and Management（巴拉里技术与管理学院）
In real-world environments, AI systems often face unfamiliar scenarios without labeled data, creating a major challenge for conventional scene understanding models. The inability to generalize across unseen contexts limits the deployment of vision-based applications in dynamic, unstructured settings. This work introduces a Dynamic Context-Aware Scene Reasoning framework that leverages Vision-Language Alignment to address zero-shot real-world scenarios. The goal is to enable intelligent systems to infer and adapt to new environments without prior task-specific training. The proposed approach in…
Vision-Language Pre-training with Object Contrastive Learning for 3D Scene Understanding
Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；College of Computer Science and Software Engineering, Shenzhen University（深圳大学计算机与软件学院）；Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳））；Research Center of Artificial Intelligence, Peng Cheng Laboratory（鹏城实验室人工智能研究中心）
From Pixels to Graphs: Open-Vocabulary Scene Graph Generation with Vision-Language Models
School of Information Science and Technology, ShanghaiTech University（上海科技大学信息科学与技术学院）；Shanghai AI Laboratory（上海人工智能实验室）；Shanghai Engineering Research Center of Intelligent Vision and Imaging（上海智能视觉与成像工程技术研究中心）
Autonomous spacecraft relative navigation technology has been planned for and applied to many famous space missions. The development of on-board electronics systems has enabled the use of vision-based and LiDAR-based methods to achieve better performances. Meanwhile, deep learning has reached great success in different areas, especially in computer vision, which has also attracted the attention of space researchers. However, spacecraft navigation differs from ground tasks due to high reliability requirements but lack of large datasets. This survey aims to systematically investigate the current…
Scene text recognition (STR) enables computers to recognize and read the text in various real-world scenes. Recent STR models benefit from taking linguistic information in addition to visual cues into consideration. We propose a novel Masked Vision-Language Transformers (MVLT) to capture both the explicit and the implicit linguistic information. Our encoder is a Vision Transformer, and our decoder is a multi-modal Transformer. MVLT is trained in two stages: in the first stage, we design a STR-tailored pretraining method based on a masking strategy; in the second stage, we fine-tune our model a…
Beijing Institute of Technology（北京理工大学）；The State Key Laboratory for Management and Control of Complex Systems, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所复杂系统管理与控制国家重点实验室）；Qingdao Academy of Intelligent Industries（青岛智能产业研究院）；University of Science and Technology of China（中国科学技术大学）
Video image datasets are playing an essential role in design and evaluation of traffic vision algorithms. Nevertheless, a longstanding inconvenience concerning image datasets is that manually collecting and annotating large-scale diversified datasets from real scenes is time-consuming and prone to error. For that virtual datasets have begun to function as a proxy of real datasets. In this paper, we propose to construct large-scale artificial scenes for traffic vision research and generate a new virtual dataset called "ParallelEye". First of all, the street map data is used to build 3D scene mo…
Wangxuan Institute of Computer Technology Peking University, Beijing, China Email
Scene text recognition (STR) is a challenging problem due to the imperfect imagery conditions in natural images. State-of-the-art methods utilize both visual cues and linguistic knowledge to tackle this challenging problem. Specifically, they propose iterative language modeling module (IterLM) to repeatedly refine the output sequence from the visual modeling module (VM). Though achieving promising results, the vision modeling module has become the performance bottleneck of these methods. In this paper, we newly propose iterative vision modeling module (IterVM) to further improve the STR accura…
We present a Polyhedral Scene Generator system which creates a random scene based on a few user parameters, renders the scene from random view points and creates a dataset containing the renderings and corresponding annotation files. We hope that this generator will enable research on how a program could parse a scene if it had multiple viewpoints to consider. For ambiguous scenes, typically people move their head or change their position to see the scene from different angles as well as seeing how it changes while they move; this research field is called active perception. The random scene ge…
Facebook AI Research（脸书人工智能研究院）；McGill University（麦吉尔大学）
Multi-view implicit scene reconstruction methods have become increasingly popular due to their ability to represent complex scene details. Recent efforts have been devoted to improving the representation of input information and to reducing the number of views required to obtain high quality reconstructions. Yet, perhaps surprisingly, the study of which views to select to maximally improve scene understanding remains largely unexplored. We propose an uncertainty-driven active vision approach for implicit scene reconstruction, which leverages occupancy uncertainty accumulated across the scene u…
Harbin Institute of Technology（哈尔滨工业大学）；China Academy of Space Technology（中国空间技术研究院）
Perceiving the three-dimensional (3D) structure of the spacecraft is a prerequisite for successfully executing many on-orbit space missions, and it can provide critical input for many downstream vision algorithms. In this paper, we propose to sense the 3D structure of spacecraft using light detection and ranging sensor (LIDAR) and a monocular camera. To this end, Spacecraft Depth Completion Network (SDCNet) is proposed to recover the dense depth map based on gray image and sparse depth map. Specifically, SDCNet decomposes the object-level spacecraft depth completion task into foreground segmen…
Georgia Institute of Technology（佐治亚理工学院）；Oregon State University（俄勒冈州立大学）
Natural language instructions for visual navigation often use scene descriptions (e.g., "bedroom") and object references (e.g., "green chairs") to provide a breadcrumb trail to a goal location. This work presents a transformer-based vision-and-language navigation (VLN) agent that uses two different visual encoders -- a scene classification network and an object detector -- which produce features that match these two distinct types of visual cues. In our method, scene features contribute high-level contextual information that supports object-level processing. With this design, our model is able…
The driving behavior at urban intersections is very complex. It is thus crucial for autonomous vehicles to comprehensively understand challenging urban traffic scenes in order to navigate intersections and prevent accidents. In this paper, we introduce a stereo vision and 3D digital map based approach to spatially and temporally analyze the traffic situation at urban intersections. Stereo vision is used to detect, classify and track obstacles, while a 3D digital map is used to improve ego-localization and provide context in terms of road-layout information. A probabilistic approach that tempor…
The State Key Laboratory for Management and Control of Complex Systems, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所复杂系统管理与控制国家重点实验室）；Qingdao Academy of Intelligent Industries（青岛智能产业研究院）；Research Center for Computational Experiments and Parallel Systems Technology, National University of Defense Technology（国防科技大学计算实验与平行系统技术研究中心）
As a special type of object detection, pedestrian detection in generic scenes has made a significant progress trained with large amounts of labeled training data manually. While the models trained with generic dataset work bad when they are directly used in specific scenes. With special viewpoints, flow light and backgrounds, datasets from specific scenes are much different from the datasets from generic scenes. In order to make the generic scene pedestrian detectors work well in specific scenes, the labeled data from specific scenes are needed to adapt the models to the specific scenes. While…
LEO-VL: Efficient Scene Representation for Scalable 3D Vision-Language Learning
Institute for Artificial Intelligence, Peking University（北京大学人工智能研究院）；State Key Laboratory of General Artificial Intelligence, BIGAI（通用人工智能国家重点实验室，BIGAI）；School of Artificial Intelligence, Beijing University of Posts and Telecommunications（北京邮电大学人工智能学院）；Department of Automation, Tsinghua University（清华大学自动化系）
MAPS: A Synthetic Dataset for Probing Vision Models in a Controlled 3D Scene Space
FIAS & Institute of Computer Science Goethe University Frankfurt（FIAS与计算机科学研究所弗赖堡大学）；Mila & Department of Biology York University（Mila与生物学系约克大学）；Institute of Computer Science Goethe University Frankfurt（计算机科学研究所弗赖堡大学）
