There is a general expectation that robots should operate in environments that consist of static and dynamic entities including people, furniture and automobiles. These dynamic environments pose challenges to visual simultaneous localization and mapping (SLAM) algorithms by introducing errors into the front-end. Light fields provide one possible method for addressing such problems by capturing a more complete visual information of a scene. In contrast to a single ray from a perspective camera, Light Fields capture a bundle of light rays emerging from a single point in space, allowing us to see through dynamic objects by refocusing past them.
In this paper we present a method to synthesize a refocused image of the static background in the presence of dynamic objects that uses a light-field acquired with a linear camera array. We simultaneously estimate both the depth and the refocused image of the static scene using semantic segmentation for detecting dynamic objects in a single time step. This eliminates the need for initializing a static map . The algorithm is parallelizable and is implemented on GPU allowing us execute it at close to real time speeds. We demonstrate the effectiveness of our method on real-world data acquired using a small robot with a five camera array.
POV-SLAM: Probabilistic Object-Aware Variational SLAM in Semi-Static Environments
The University of Toronto Institute for Aerospace Studies（多伦多大学航空航天研究所）；The Technical University of Munich（慕尼黑工业大学）；Clearpath Robotics（Clearpath机器人公司）；The Technical University of Nuremberg（纽伦堡工业大学）
College of Information Science and Engineering, Northeastern University（东北大学信息科学与工程学院）
Visual slam technology is one of the key technologies for robot to explore unknown environment independently. Accurate estimation of camera pose based on visual sensor is the basis of autonomous navigation and positioning. However, most visual slam algorithms are based on static environment assumption and cannot estimate accurate camera pose in dynamic environment. In order to solve this problem, a visual SLAM algorithm for indoor dynamic environment is proposed. Firstly, some moving objects are eliminated based on the depth information of RGB-D camera, and the initial camera pose is obtained …
Inspired by the recent success of application of dense data approach by using ORB-SLAM and RGB-D SLAM, we propose a better pipeline of real-time SLAM in dynamics environment. Different from previous SLAM which can only handle static scenes, we are presenting a solution which use RGB-D SLAM as well as YOLO real-time object detection to segment and remove dynamic scene and then construct static scene 3D. We gathered a dataset which allows us to jointly consider semantics, geometry, and physics and thus enables us to reconstruct the static scene while filtering out all dynamic objects.
We address the problem of detecting and erasing furniture from a wide angle photograph of a room. Inpainting large regions of an indoor scene often results in geometric inconsistencies of background elements within the inpaint mask. To address this problem, we utilize perceptual information (e.g. instance segmentation, and room layout) to produce a geometrically consistent empty version of a room. We share important details to make this system viable, such as per-plane inpainting, automatic rectification, and texture refinement. We provide detailed ablation along with qualitative examples, jus…
Department of Automation, Shanghai Jiao Tong University（自动化系，上海交通大学）；School of Information Science and Electronic Engineering, Shanghai Jiao Tong University（信息科学与电子工程学院，上海交通大学）；Bosch（博世）；School of Mechanical and Aerospace Engineering, Nanyang Technological University（机械与航空航天工程学院，南洋理工大学）；Department of Automation, Key Laboratory of System Control and Information Processing of Ministry of Education, State Key Laboratory of Avionics Integration and Aviation System-of-Systems Synthesis, Shanghai Key Laboratory of Navigation and Location Based Services, Shanghai Jiao Tong University（自动化系，教育部系统控制与信息处理重点实验室，航空系统集成与航空系统-of-系统综合国家重点实验室，导航与位置基于服务上海市重点实验室，上海交通大学）
Manipulating images of complex scenes to reconstruct, insert and/or remove specific object instances is a challenging task. Complex scenes contain multiple semantics and objects, which are frequently cluttered or ambiguous, thus hampering the performance of inpainting models. Conventional techniques often rely on structural information such as object contours in multi-stage approaches that generate unreliable results and boundaries. In this work, we propose a novel deep learning model to alter a complex urban scene by removing a user-specified portion of the image and coherently inserting a ne…
University of Edinburgh（爱丁堡大学）；Shenzhen Institute of Artificial Intelligence and Robotics for Society（深圳市人工智能与机器人研究院）
This work presents a novel RGB-D SLAM approach to simultaneously segment, track and reconstruct the static background and large dynamic rigid objects that can occlude major portions of the camera view. Previous approaches treat dynamic parts of a scene as outliers and are thus limited to a small amount of changes in the scene, or rely on prior information for all objects in the scene to enable robust camera tracking. Here, we propose to treat all dynamic parts as one rigid body and simultaneously segment and track both static and dynamic components. We, therefore, enable simultaneous localisat…
Simon Fraser University（西蒙弗雷泽大学）；Beihang University（北京航空航天大学）；Jilin University（吉林大学）
We introduce RaidaR, a rich annotated image dataset of rainy street scenes, to support autonomous driving research. The new dataset contains the largest number of rainy images (58,542) to date, 5,000 of which provide semantic segmentations and 3,658 provide object instance segmentations. The RaidaR images cover a wide range of realistic rain-induced artifacts, including fog, droplets, and road reflections, which can effectively augment existing street scene datasets to improve data-driven machine perception during rainy weather. To facilitate efficient annotation of a large volume of images, w…
Predicting the trajectories of surrounding agents is an essential ability for autonomous vehicles navigating through complex traffic scenes. The future trajectories of agents can be inferred using two important cues: the locations and past motion of agents, and the static scene structure. Due to the high variability in scene structure and agent configurations, prior work has employed the attention mechanism, applied separately to the scene and agent configuration to learn the most salient parts of both cues. However, the two cues are tightly linked. The agent configuration can inform what part…
University of Toronto Robotics Institute（多伦多大学机器人研究所）；Clearpath Robotics（Clearpath机器人公司）
Maintaining an up-to-date map to reflect recent changes in the scene is very important, particularly in situations involving repeated traversals by a robot operating in an environment over an extended period. Undetected changes may cause a deterioration in map quality, leading to poor localization, inefficient operations, and lost robots. Volumetric methods, such as truncated signed distance functions (TSDFs), have quickly gained traction due to their real-time production of a dense and detailed map, though map updating in scenes that change over time remains a challenge. We propose a framewor…
Robotic Systems Lab, ETH Zürich（苏黎世联邦理工学院机器人系统实验室）
We present a model predictive controller (MPC) that automatically discovers collision-free locomotion while simultaneously taking into account the system dynamics, friction constraints, and kinematic limitations. A relaxed barrier function is added to the optimization's cost function, leading to collision avoidance behavior without increasing the problem's computational complexity. Our holistic approach does not require any heuristics and enables legged robots to find whole-body motions in the presence of static and dynamic obstacles. We use a dynamically generated euclidean signed distance fi…
Predictive Spatio-Temporal Scene Graphs for Semi-Static Scenes
Department of Computer Science and Operations Research, Université de Montréal（蒙特利尔大学计算机科学与运筹学系）；Mila - Quebec AI Institute（魁北克人工智能研究所）；Hydro-Québec Research Institute (IREQ)（魁北克水电研究院（IREQ））
We present a new approach for redirected walking in static and dynamic scenes that uses techniques from robot motion planning to compute the redirection gains that steer the user on collision-free paths in the physical space. Our first contribution is a mathematical framework for redirected walking using concepts from motion planning and configuration spaces. This framework highlights various geometric and perceptual constraints that tend to make collision-free redirected walking difficult. We use our framework to propose an efficient solution to the redirection problem that uses the notion of…
Woods Hole Oceanographic Institution（伍兹霍尔海洋研究所）；Point Loma Nazarene University（波因特洛马拿撒勒大学）；NOAA（美国国家海洋和大气管理局）
This paper explores the use of a Bayesian non-parametric topic modeling technique for the purpose of anomaly detection in video data. We present results from two experiments. The first experiment shows that the proposed technique is automatically able characterize the underlying terrain, and detect anomalous flora in image data collected by an underwater robot. The second experiment shows that the same technique can be used on images from a static camera in a dynamic unstructured environment. In the second dataset, consisting of video data from a static seafloor camera capturing images of a bu…
You Only Scan Once: A Dynamic Scene Reconstruction Pipeline for 6-DoF Robotic Grasping of Novel Objects
Advanced Robotics Centre, National University of Singapore（新加坡国立大学先进机器人中心）；Integrative Sciences and Engineering Programme, National University of Singapore Graduate School（新加坡国立大学研究生院整合科学与工程项目）
We present a system for applying sim2real approaches to "in the wild" scenes with realistic visuals, and to policies which rely on active perception using RGB cameras. Given a short video of a static scene collected using a generic phone, we learn the scene's contact geometry and a function for novel view synthesis using a Neural Radiance Field (NeRF). We augment the NeRF rendering of the static scene by overlaying the rendering of other dynamic objects (e.g. the robot's own body, a ball). A simulation is then created using the rendering engine in a physics simulator which computes contact dyn…
Event-guided 3D Gaussian Splatting for Dynamic Human and Scene Reconstruction
College of Optical Science and Engineering, Zhejiang University, China（浙江大学光学工程学院）；School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore（南洋理工大学电子与电气工程学院）；School of Artificial Intelligence and Robotics, Hunan University, China（湖南大学人工智能与机器人学院）
THUD++: Large-Scale Dynamic Indoor Scene Dataset and Benchmark for Mobile Robots
Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；Department of Computer Science and Technology, Tsinghua University（清华大学计算机科学与技术系）
