Moving Object Detection (MOD) is an important task for achieving robust autonomous driving. An autonomous vehicle has to estimate collision risk with other interacting objects in the environment and calculate an optional trajectory. Collision risk is typically higher for moving objects than static ones due to the need to estimate the future states and poses of the objects for decision making. This is particularly important for near-range objects around the vehicle which are typically detected by a fisheye surround-view system that captures a 360° view of the scene. In this work, we propose a CNN architecture for moving object detection using fisheye images that were captured in autonomous driving environment. As motion geometry is highly non-linear and unique for fisheye cameras, we will make an improved version of the current dataset public to encourage further research. To target embedded deployment, we design a lightweight encoder sharing weights across sequential images. The proposed network runs at 15 fps on a 1 teraflops automotive embedded system at accuracy of 40% IoU and 69.5% mIoU.
For human drivers, having rear and side-view mirrors is vital for safe driving. They deliver a more complete view of what is happening around the car. Human drivers also heavily exploit their mental map for navigation. Nonetheless, several methods have been published that learn driving models with only a front-facing camera and without a route planner. This lack of information renders the self-driving task quite intractable. We investigate the problem in a more realistic setting, which consists of a surround-view camera system with eight cameras, a route planner, and a CAN bus reader. In parti…
Semi-SMD: Semi-Supervised Metric Depth Estimation via Surrounding Cameras for Autonomous Driving
Robotics and Autonomous Systems Thrust, The Hong Kong University of Science and Technology (Guangzhou)（机器人与自主系统方向，香港科学与技术大学（广州））；Department of Electronic and Computer Engineering, The Hong Kong University of Science and Technology（电子与计算机工程系，香港科学与技术大学）
Zongmu Technology（纵目科技）；University of Victoria（维多利亚大学）；Xiamen University（厦门大学）
The vehicle re-identification (ReID) plays a critical role in the perception system of autonomous driving, which attracts more and more attention in recent years. However, to our best knowledge, there is no existing complete solution for the surround-view system mounted on the vehicle. In this paper, we argue two main challenges in above scenario: i) In single camera view, it is difficult to recognize the same vehicle from the past image frames due to the fisheye distortion, occlusion, truncation, etc. ii) In multi-camera view, the appearance of the same vehicle varies greatly from different c…
In this research we present a novel algorithm for background subtraction using a moving camera. Our algorithm is based purely on visual information obtained from a camera mounted on an electric bus, operating in downtown Reno which automatically detects moving objects of interest with the view to provide a fully autonomous vehicle. In our approach we exploit the optical flow vectors generated by the motion of the camera while keeping parameter assumptions a minimum. At first, we estimate the Focus of Expansion, which is used to model and simulate 3D points given the intrinsic parameters of the…
Surround-view cameras are a primary sensor for automated driving, used for near-field perception. It is one of the most commonly used sensors in commercial vehicles primarily used for parking visualization and automated parking. Four fisheye cameras with a 190° field of view cover the 360° around the vehicle. Due to its high radial distortion, the standard algorithms do not extend easily. Previously, we released the first public fisheye surround-view dataset named WoodScape. In this work, we release a synthetic version of the surround-view dataset, covering many of its weaknesses and extending…
In this paper, we introduce a moving object detection algorithm for fisheye cameras used in autonomous driving. We reformulate the three commonly used constraints in rectilinear images (epipolar, positive depth and positive height constraints) to spherical coordinates which is invariant to specific camera configuration once the calibration is known. One of the main challenging use case in autonomous driving is to detect parallel moving objects which suffer from motion-parallax ambiguity. To alleviate this, we formulate an additional fourth constraint, called the anti-parallel constraint, which…
Zongmu Technology（纵目科技）；Technical University of Braunschweig（布伦瑞克工业大学）
Vehicle re-identification (Re-ID) is a critical component of the autonomous driving perception system, and research in this area has accelerated in recent years. However, there is yet no perfect solution to the vehicle re-identification issue associated with the car's surround-view camera system. Our analysis identifies two significant issues in the aforementioned scenario: i) It is difficult to identify the same vehicle in many picture frames due to the unique construction of the fisheye camera. ii) The appearance of the same vehicle when seen via the surround vision system's several cameras …
Surround-View Fisheye Optics in Computer Vision and Simulation: Survey and Challenges
University of Limerick（利默里克大学）；University of Galway（戈尔韦大学）；Lero (the Science Foundation Ireland Research Centre for Software)（Lero（爱尔兰科学基金会软件研究中心））；Valeo Vision Systems（法雷奥视觉系统公司）
Moving object detection is a critical task for autonomous vehicles. As dynamic objects represent higher collision risk than static ones, our own ego-trajectories have to be planned attending to the future states of the moving elements of the scene. Motion can be perceived using temporal information such as optical flow. Conventional optical flow computation is based on camera sensors only, which makes it prone to failure in conditions with low illumination. On the other hand, LiDAR sensors are independent of illumination, as they measure the time-of-flight of their own emitted lasers. In this …
Driving requires interacting with road agents and predicting their future behaviour in order to navigate safely. We present FIERY: a probabilistic future prediction model in bird's-eye view from monocular cameras. Our model predicts future instance segmentation and motion of dynamic agents that can be transformed into non-parametric future trajectories. Our approach combines the perception, sensor fusion and prediction components of a traditional autonomous driving stack by estimating bird's-eye-view prediction directly from surround RGB monocular camera inputs. FIERY learns to model the inher…
Zongmu Technology（纵目科技）；Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）
The 3D visual perception for vehicles with the surround-view fisheye camera system is a critical and challenging task for low-cost urban autonomous driving. While existing monocular 3D object detection methods perform not well enough on the fisheye images for mass production, partly due to the lack of 3D datasets of such images. In this paper, we manage to overcome and avoid the difficulty of acquiring the large scale of accurate 3D labeled truth data, by breaking down the 3D object detection task into some sub-tasks, such as vehicle's contact point detection, type classification, re-identific…
FishBEV: Distortion-Resilient Bird's Eye View Segmentation with Surround-View Fisheye Cameras
College of Computer Science, Nankai University（南开大学计算机科学学院）；Key Laboratory of Data and Intelligent System Security, Ministry of Education, China（教育部数据与智能系统安全重点实验室）；School of Cyber Security, University of Science and Technology of China（中国科学技术大学网络安全学院）；Haihe Lab of ITAI, Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所海河实验室）
SurroundOcc: Multi-Camera 3D Occupancy Prediction for Autonomous Driving
Beijing National Research Center for Information Science and Technology（北京信息科学与技术国家研究中心）；Tsinghua University（清华大学）；Tianjin University（天津大学）；PhiGent Robotics（鉴智机器人）
OccTrack360: 4D Panoptic Occupancy Tracking from Surround-View Fisheye Cameras
School of Artificial Intelligence and Robotics, Hunan University（人工智能与机器人学院，湖南大学）；State Key Laboratory of Extreme Photonics and Instrumentation, Zhejiang University（极端光子学与仪器国家重点实验室，浙江大学）；National Engineering Research Center of Robot Visual Perception and Control Technology, Hunan University（机器人视觉感知与控制技术国家工程研究中心，湖南大学）
In the emerging advancement in the branch of autonomous robotics, the ability of a robot to efficiently localize and construct maps of its surrounding is crucial. This paper deals with utilizing thermal-infrared cameras, as opposed to conventional cameras as the primary sensor to capture images of the robot's surroundings. For localization, the images need to be further processed before feeding them to a navigational system. The main motivation of this paper was to develop an edge detection methodology capable of utilizing the low-SNR poor output from such a thermal camera and effectively dete…
We present ClusterVO, a stereo Visual Odometry which simultaneously clusters and estimates the motion of both ego and surrounding rigid clusters/objects. Unlike previous solutions relying on batch input or imposing priors on scene structure or dynamic object models, ClusterVO is online, general and thus can be used in various scenarios including indoor scene understanding and autonomous driving. At the core of our system lies a multi-level probabilistic association mechanism and a heterogeneous Conditional Random Field (CRF) clustering approach combining semantic, spatial and motion informatio…
Robotics Research Center, KCIS, IIIT Hyderabad（印度海得拉巴国际信息技术研究所 KCIS 机器人研究中心）；Montreal Institute of Learning Algorithms (MILA), Universite de Montreal（蒙特利尔大学蒙特利尔学习算法研究所）
Accurate localization of other traffic participants is a vital task in autonomous driving systems. State-of-the-art systems employ a combination of sensing modalities such as RGB cameras and LiDARs for localizing traffic participants, but most such demonstrations have been confined to plain roads. We demonstrate, to the best of our knowledge, the first results for monocular object localization and shape estimation on surfaces that do not share the same plane with the moving monocular camera. We approximate road surfaces by local planar patches and use semantic cues from vehicles in the scene t…
The formation of eyes led to the big bang of evolution. The dynamics changed from a primitive organism waiting for the food to come into contact for eating food being sought after by visual sensors. The human eye is one of the most sophisticated developments of evolution, but it still has defects. Humans have evolved a biological perception algorithm capable of driving cars, operating machinery, piloting aircraft, and navigating ships over millions of years. Automating these capabilities for computers is critical for various applications, including self-driving cars, augmented reality, and arc…
South China University of Technology（华南理工大学）；Pazhou Laboratory（琶洲实验室）；Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学高瓴人工智能学院）；Information Technology R&D Innovation Center of Peking University（北京大学信息技术研发创新中心）；MIT-IBM Watson AI Lab（麻省理工学院-IBM沃森人工智能实验室）；UMass Amherst（马萨诸塞大学阿默斯特分校）
Getting robots to navigate to multiple objects autonomously is essential yet difficult in robot applications. One of the key challenges is how to explore environments efficiently with camera sensors only. Existing navigation methods mainly focus on fixed cameras and few attempts have been made to navigate with active cameras. As a result, the agent may take a very long time to perceive the environment due to limited camera scope. In contrast, humans typically gain a larger field of view by looking around for a better perception of the environment. How to make robots perceive the environment as…
University of California Berkeley（加州大学伯克利分校）；DSO National Laboratories（新加坡国防科技研究院）；National University of Singapore（新加坡国立大学）；Graz University of Technology（格拉茨技术大学）；ETH Zürich（苏黎世联邦理工学院）；Microsoft（微软）
Cameras are a crucial exteroceptive sensor for self-driving cars as they are low-cost and small, provide appearance information about the environment, and work in various weather conditions. They can be used for multiple purposes such as visual navigation and obstacle detection. We can use a surround multi-camera system to cover the full 360-degree field-of-view around the car. In this way, we avoid blind spots which can otherwise lead to accidents. To minimize the number of cameras needed for surround perception, we utilize fisheye cameras. Consequently, standard vision pipelines for 3D mappi…
Detection of moving objects is a very important task in autonomous driving systems. After the perception phase, motion planning is typically performed in Bird's Eye View (BEV) space. This would require projection of objects detected on the image plane to top view BEV plane. Such a projection is prone to errors due to lack of depth information and noisy mapping in far away areas. CNNs can leverage the global context in the scene to project better. In this work, we explore end-to-end Moving Object Detection (MOD) on the BEV map directly using monocular images as input. To the best of our knowled…
Cam4DOcc: Benchmark for Camera-Only 4D Occupancy Forecasting in Autonomous Driving Applications
Shanghai Jiao Tong University（上海交通大学）；HAOMO.AI Technology Co., Ltd.（HAOMO.AI技术有限公司）；National University of Defense Technology（国防科技大学）；Beijing Institute of Technology（北京理工大学）
MetaOcc: Spatio-Temporal Fusion of Surround-View 4D Radar and Camera for 3D Occupancy Prediction with Dual Training Strategies
School of Automotive Studies, Tongji University（同济大学汽车学院）；College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；School of Automobile, Chang'an University（长安大学汽车学院）；School of Information and Electrical Engineering, Hangzhou City University（杭州城市学院信息与电气工程学院）；College of Science and Engineering, James Cook University（詹姆斯库克大学科学与工程学院）
This paper presents a new high resolution aerial images dataset in which moving objects are labelled manually. It aims to contribute to the evaluation of the moving object detection methods for moving cameras. The problem of recognizing moving objects from aerial images is one of the important issues in computer vision. The biggest problem in the images taken by UAV is that the background is constantly variable due to camera movement. There are various datasets in the literature in which proposed methods for motion detection are evaluated. Prepared dataset consists of challenging images contai…
During about 30 years, a lot of research teams have worked on the big challenge of detection of moving objects in various challenging environments. First applications concern static cameras but with the rise of the mobile sensors studies on moving cameras have emerged over time. In this survey, we propose to identify and categorize the different existing methods found in the literature. For this purpose, we propose to classify these methods according to the choose of the scene representation: one plane or several parts. Inside these two categories, the methods are grouped according to eight di…
Moving object detection is important in computer vision. Event-based cameras are bio-inspired cameras that work by mimicking the working of the human eye. These cameras have multiple advantages over conventional frame-based cameras, like reduced latency, HDR, reduced motion blur during high motion, low power consumption, etc. In spite of these advantages, event-based cameras are noise-sensitive and have low resolution. Moreover, the task of moving object detection in these cameras is difficult, as event-based sensors lack useful visual features like texture and color. In this paper, we investi…
Beijing National Research Center for Information Science and Technology（北京信息科学与技术国家研究中心）；Tsinghua University（清华大学）；Tianjin University（天津大学）；PhiGent Robotics（鉴智机器人）
Depth estimation from images serves as the fundamental step of 3D perception for autonomous driving and is an economical alternative to expensive depth sensors like LiDAR. The temporal photometric constraints enables self-supervised depth estimation without labels, further facilitating its application. However, most existing methods predict the depth solely based on each monocular image and ignore the correlations among multiple surrounding cameras, which are typically available for modern self-driving vehicles. In this paper, we propose a SurroundDepth method to incorporate the information fr…
Surround-view fisheye cameras are commonly used for near-field sensing in automated driving. Four fisheye cameras on four sides of the vehicle are sufficient to cover 360° around the vehicle capturing the entire near-field region. Some primary use cases are automated parking, traffic jam assist, and urban driving. There are limited datasets and very little work on near-field perception tasks as the focus in automotive perception is on far-field perception. In contrast to far-field, surround-view perception poses additional challenges due to high precision object detection requirements of 10cm …
Shanghai Jiao Tong University（上海交通大学）；SJTU-ParisTech Elite Institute of Technology（上海交通大学-巴黎高科卓越工程学院）
Understanding the surrounding environment of the vehicle is still one of the challenges for autonomous driving. This paper addresses 360-degree road scene semantic segmentation using surround view cameras, which are widely equipped in existing production cars. First, in order to address large distortion problem in the fisheye images, Restricted Deformable Convolution (RDC) is proposed for semantic segmentation, which can effectively model geometric transformations by learning the shapes of convolutional filters conditioned on the input feature map. Second, in order to obtain a large-scale trai…
Understanding ego-motion and surrounding vehicle state is essential to enable automated driving and advanced driving assistance technologies. Typical approaches to solve this problem use fusion of multiple sensors such as LiDAR, camera, and radar to recognize surrounding vehicle state, including position, velocity, and orientation. Such sensing modalities are overly complex and costly for production of personal use vehicles. In this paper, we propose a novel machine learning method to estimate ego-motion and surrounding vehicle state using a single monocular camera. Our approach is based on a …
Towards Effective Multi-Moving-Camera Tracking: A New Dataset and Lightweight Link Model
School of Computer Science and Technology, Donghua University（东华大学计算机科学与技术学院）；Institute of Integrated Science and Technology, Hosei University（法政大学综合科学技术研究所）；Frontiers Science Center for Intelligent Autonomous Systems, Tongji University（同济大学智能自主系统前沿科学中心）
To detect saliency in video is a fundamental step in many computer vision systems. Saliency is the significant target(s) in the video. The object of interest is further analyzed for high-level applications. The segregation of saliency and the background can be made if they exhibit different visual cues. Therefore, saliency detection is often formulated as background subtraction. However, saliency detection is challenging. For instance, dynamic background can result in false positive errors. In another scenario, camouflage will lead to false negative errors. With moving camera, the captured sce…
Toyota Research Institute（丰田研究院）；Toyota Technological Institute at Chicago（芝加哥丰田技术学院）
Self-supervised monocular depth and ego-motion estimation is a promising approach to replace or supplement expensive depth sensors such as LiDAR for robotics applications like autonomous driving. However, most research in this area focuses on a single monocular camera or stereo pairs that cover only a fraction of the scene around the vehicle. In this work, we extend monocular self-supervised depth and ego-motion estimation to large-baseline multi-camera rigs. Using generalized spatio-temporal contexts, pose consistency constraints, and carefully designed photometric loss masking, we learn a si…
Inst. of Neuroinformatics, UZH-ETH ZUrich（苏黎世大学-苏黎世联邦理工学院神经信息学研究所）
Standard dynamic vision sensor (DVS) event cameras output a stream of spatially-independent log-intensity brightness change events so they cannot suppress spatial redundancy. Nearly all biological retinas use an antagonistic center-surround organization. This paper proposes a practical method of implementing a compact, energy-efficient Center Surround DVS (CSDVS) with a surround smoothing network that uses compact polysilicon resistors for lateral resistance. The paper includes behavioral simulation results for the CSDVS (see sites.google.com/view/csdvs/home). The CSDVS would significantly red…
Camera calibration for the surround-view system: a benchmark and dataset
Institute of Information Science, Beijing Jiaotong University（北京交通大学信息科学研究所）；Beijing Key Laboratory of Advanced Information Science and Network（北京先进信息科学与网络重点实验室）
Moving Object Detection from Moving Camera Using Focus of Expansion Likelihood and Segmentation
Department of Precision Engineering, Graduate School of Engineering, The University of Tokyo（精密工程学系，工学研究生院，东京大学）；Department of Human and Engineered Environmental Studies, Graduate School of Frontier Sciences, The University of Tokyo（人类与工程环境研究系，前沿科学研究生院，东京大学）
Valeo DAR Kronach（法雷奥DAR克罗纳赫）；TU Ilmenau（伊尔梅瑙工业大学）；TU Braunschweig（布伦瑞克工业大学）；Valeo Vision Systems（法雷奥视觉系统）
A 360° perception of scene geometry is essential for automated driving, notably for parking and urban driving scenarios. Typically, it is achieved using surround-view fisheye cameras, focusing on the near-field area around the vehicle. The majority of current depth estimation approaches focus on employing just a single camera, which cannot be straightforwardly generalized to multiple cameras. The depth estimation model must be tested on a variety of cameras equipped to millions of cars with varying camera geometries. Even within a single car, intrinsics vary due to manufacturing tolerances. De…
École Polytechnique Fédérale de Lausanne (EPFL)（洛桑联邦理工学院）；Graz University of Technology（格拉茨工业大学）
We propose an approach to detect flying objects such as UAVs and aircrafts when they occupy a small portion of the field of view, possibly moving against complex backgrounds, and are filmed by a camera that itself moves. Solving such a difficult problem requires combining both appearance and motion cues. To this end we propose a regression-based approach to motion stabilization of local image patches that allows us to achieve effective classification on spatio-temporal image cubes and outperform state-of-the-art techniques. As the problem is relatively new, we collected two challenging dataset…
The exponentially increasing use of moving platforms for video capture introduces the urgent need to develop the general background subtraction algorithms with the capability to deal with the moving background. In this paper, we propose a multilayer-based framework for online background subtraction for videos captured by moving cameras. Unlike the previous treatments of the problem, the proposed method is not restricted to binary segmentation of background and foreground, but formulates it as a multi-label segmentation problem by modeling multiple foreground objects in different layers when th…
3D Trajectory Reconstruction of Moving Points Based on Asynchronous Cameras
College of Aerospace Science and Engineering, National University of Defense Technology（航空科学与工程学院，国防科技大学）；Hunan Provincial Key Laboratory of Image Measurement and Vision Navigation（湖南省图像测量与视觉导航重点实验室）
