Object detection is essential to safe autonomous or assisted driving. Previous works usually utilize RGB images or LiDAR point clouds to identify and localize multiple objects in self-driving. However, cameras tend to fail in bad driving conditions, e.g. bad weather or weak lighting, while LiDAR scanners are too expensive to get widely deployed in commercial applications. Radar has been drawing more and more attention due to its robustness and low cost. In this paper, we propose a scene-aware radar learning framework for accurate and robust object detection. First, the learning framework contains branches conditioning on the scene category of the radar sequence; with each branch optimized for a specific type of scene. Second, three different 3D autoencoder-based architectures are proposed for radar object detection and ensemble learning is performed over the different architectures to further boost the final performance. Third, we propose novel scene-aware sequence mix augmentation (SceneMix) and scene-specific post-processing to generate more robust detection results. In the ROD2021 Challenge, we achieved a final result of average precision of 75.0% and an average recall of 81.0%. Moreover, in the parking lot scene, our framework ranks first with an average precision of 97.8% and an average recall of 98.6%, which demonstrates the effectiveness of our framework.
General Motors, Advanced Technical Center（通用汽车先进技术中心）
Occupancy grid mapping is an important component in road scene understanding for autonomous driving. It encapsulates information of the drivable area, road obstacles and enables safe autonomous driving. Radars are an emerging sensor in autonomous vehicle vision, becoming more widely used due to their long range sensing, low cost, and robustness to severe weather conditions. Despite recent advances in deep learning technology, occupancy grid mapping from radar data is still mostly done using classical filtering approaches.In this work, we propose learning the inverse sensor model used for occup…
Autonomous driving requires a detailed understanding of complex driving scenes. The redundancy and complementarity of the vehicle's sensors provide an accurate and robust comprehension of the environment, thereby increasing the level of performance and safety. This thesis focuses the on automotive RADAR, which is a low-cost active sensor measuring properties of surrounding objects, including their relative speed, and has the key advantage of not being impacted by adverse weather conditions. With the rapid progress of deep learning and the availability of public driving datasets, the perception…
LTCI, Télécom Paris, Institut Polytechnique de Paris（巴黎综合理工学院电信学院LTCI实验室）
Understanding the scene around the ego-vehicle is key to assisted and autonomous driving. Nowadays, this is mostly conducted using cameras and laser scanners, despite their reduced performances in adverse weather conditions. Automotive radars are low-cost active sensors that measure properties of surrounding objects, including their relative speed, and have the key advantage of not being impacted by rain, snow or fog. However, they are seldom used for scene understanding due to the size and complexity of radar raw data and the lack of annotated datasets. Fortunately, recent open-sourced datase…
SemRaFiner: Panoptic Segmentation in Sparse and Noisy Radar Point Clouds
CARIAD SE；University of Bonn（波恩大学）；Technical University of Munich（慕尼黑工业大学）；Lamarr Institute for Machine Learning and Artificial Intelligence（拉马尔机器学习与人工智能研究所）
Boosting Instance Awareness via Cross-View Correlation with 4D Radar and Camera for 3D Object Detection
College of Information Science and Electronic Engineering, Zhejiang University（浙江大学信息科学与电子工程学院）；School of Automotive Studies, Tongji University（同济大学汽车学院）；College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；Jinhua Institute of Zhejiang University（浙江大学金华研究院）；School of Information and Electrical Engineering, Hangzhou City University（杭州城市学院信息与电气工程学院）；Hangzhou City University Binjiang Innovation Center（杭州城市学院滨江创新中心）
Sparse4D-Radar: An Efficient and Robust Framework for Surround-View 3D Object Detection via 4D Radar-Camera Fusion
State Key Laboratory of Ocean Sensing and Ocean College, Zhejiang University（浙江大学海洋传感与海洋学院海洋传感技术国家重点实验室）；Engineering Research Center of Oceanic Sensing Technology and Equipment, Ministry of Education, Zhejiang University（浙江大学海洋传感技术与装备教育部工程研究中心）；Donghai Laboratory（东海实验室）；Ocean College, Institute of Marine Electronics and Intelligent Systems, Zhejiang University（浙江大学海洋学院海洋电子与智能系统研究所）
Robust and accurate sensing is of critical importance for advancing autonomous automotive systems. The need to acquire situational awareness in complex urban conditions using sensors such as radar has motivated research on power and latency-efficient signal acquisition methods. In this paper, we present an end-to-end signal processing pipeline, capable of operating in extreme weather conditions, that relies on sub-sampled radar data to perform object detection in vehicular settings. The results of the object detection are further utilized to sub-sample forthcoming radar data, which stands in c…
Institute of Automotive Technology, Technical University of Munich（慕尼黑工业大学汽车技术研究所）；mLab:Real-Time and Embedded Systems Lab, University of Pennsylvania（宾夕法尼亚大学mLab实时与嵌入式系统实验室）
Automotive traffic scenes are complex due to the variety of possible scenarios, objects, and weather conditions that need to be handled. In contrast to more constrained environments, such as automated underground trains, automotive perception systems cannot be tailored to a narrow field of specific tasks but must handle an ever-changing environment with unforeseen events. As currently no single sensor is able to reliably perceive all relevant activity in the surroundings, sensor data fusion is applied to perceive as much information as possible. Data fusion of different sensors and sensor moda…
NavEYE: Vision-Centered Multi-Sensor Fusion-Based Situational Awareness System for Intelligent Surface Vehicles
Sanya Science and Education Innovation Park, Wuhan University of Technology（武汉理工大学三亚科教创新园）；School of Navigation, Wuhan University of Technology（武汉理工大学航运学院）；State Key Laboratory of Maritime Technology and Safety（船舶技术与安全国家重点实验室）
RADE-Net: Robust Attention Network for Radar-Only Object Detection in Adverse Weather
Infineon Technologies AG（英飞凌科技有限公司）；Institute of Visual Computing（视觉计算研究所）；Graz University of Technology（格拉茨工业大学）；Virtual Vehicle Research GmbH（虚拟车辆研究有限公司）
In this work, we present an uncertainty-based method for sensor fusion with camera and radar data. The outputs of two neural networks, one processing camera and the other one radar data, are combined in an uncertainty aware manner. To this end, we gather the outputs and corresponding meta information for both networks. For each predicted object, the gathered information is post-processed by a gradient boosting method to produce a joint prediction of both networks. In our experiments we combine the YOLOv3 object detection network with a customized $1D$ radar segmentation network and evaluate ou…
University of Oxford（牛津大学）；Oxford Robotics Institute（牛津机器人研究所）
This paper presents an efficient annotation procedure and an application thereof to end-to-end, rich semantic segmentation of the sensed environment using FMCW scanning radar. We advocate radar over the traditional sensors used for this task as it operates at longer ranges and is substantially more robust to adverse weather and illumination conditions. We avoid laborious manual labelling by exploiting the largest radar-focused urban autonomy dataset collected to date, correlating radar scans with RGB cameras and LiDAR sensors, for which semantic segmentation is an already consolidated procedur…
Weakly Supervised Cross-Modal Learning for 4D Radar Scene Flow Estimation
Zhejiang University, Hangzhou, Zhejiang, China（浙江大学）；Zhejiang Provincial Key Laboratory of Multi-Modal Communication Networks and Intelligent Information Processing（浙江省多模态通信网络与智能信息处理重点实验室）；Singapore University of Technology and Design（新加坡科技设计大学）
4DRadar-GS: Self-Supervised Dynamic Driving Scene Reconstruction with 4D Radar
School of Automotive Studies, Tongji University（同济大学汽车学院）；State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所多模态人工智能系统国家重点实验室）
University of Edinburgh（爱丁堡大学）；SAIC Motor Technical Center（上汽集团技术中心）
Scene flow allows autonomous vehicles to reason about the arbitrary motion of multiple independent objects which is the key to long-term mobile autonomy. While estimating the scene flow from LiDAR has progressed recently, it remains largely unknown how to estimate the scene flow from a 4-D radar - an increasingly popular automotive sensor for its robustness against adverse weather and lighting conditions. Compared with the LiDAR point clouds, radar data are drastically sparser, noisier and in much lower resolution. Annotated datasets for radar scene flow are also in absence and costly to acqui…
VISC: mmWave Radar Scene Flow Estimation using Pervasive Visual-Inertial Supervision
State Key Laboratory of Maritime Technology and Safety（maritime技术与安全国家重点实验室）；Wuhan University of Technology（武汉理工大学）；University of Essex（埃塞克斯大学）；Carnegie Mellon University（卡内基梅隆大学）；Tsinghua University（清华大学）；University College London（伦敦大学学院）
College of Information Science and Electronic Engineering, Zhejiang University（浙江大学信息科学与电子工程学院）；Ningbo Innovation Center, Zhejiang University（浙江大学宁波创新中心）；School of Information and Electrical Engineering, Hangzhou City University（杭州城市学院信息与电气工程学院）；Key Laboratory of Collaborative Sensing and Autonomous Unmanned Systems of Zhejiang Province（浙江省协同感知与自主无人机系统重点实验室）
ELMAR: Enhancing LiDAR Detection with 4D Radar Motion Awareness and Cross-modal Uncertainty
Infineon Technologies AG and the Technical University of Munich（英飞凌科技有限公司和技术大学慕尼黑）；China University of Geosciences（中国地质大学）；Technical University of Munich（技术大学慕尼黑）
University of Edinburgh（爱丁堡大学）；Delft University of Technology（代尔夫特理工大学）
This work proposes a novel approach to 4D radar-based scene flow estimation via cross-modal learning. Our approach is motivated by the co-located sensing redundancy in modern autonomous vehicles. Such redundancy implicitly provides various forms of supervision cues to the radar scene flow estimation. Specifically, we introduce a multi-task model architecture for the identified cross-modal learning problem and propose loss functions to opportunistically engage scene flow estimation using multiple cross-modal constraints for effective model training. Extensive experiments show the state-of-the-a…
For high resolution scene mapping and object recognition, optical technologies such as cameras and LiDAR are the sensors of choice. However, for robust future vehicle autonomy and driver assistance in adverse weather conditions, improvements in automotive radar technology, and the development of algorithms and machine learning for robust mapping and recognition are essential. In this paper, we describe a methodology based on deep neural networks to recognise objects in 300GHz radar images, investigating robustness to changes in range, orientation and different receivers in a laboratory environ…
MLF-4DRCNet: Multi-Level Fusion with 4D Radar and Camera for 3D Object Detection in Autonomous Driving
MoE Key Laboratory of Brain-Inspired Intelligence Perception and Cognition, University of Science and Technology of China（脑启发智能感知与认知教育部重点实验室，中国科学技术大学）；Institute of Artificial Intelligence, Hefei Comprehensive National Science Center（人工智能研究院，合肥国家科学中心）；Department of Electronic Engineering and Information Science, University of Science and Technology of China（电子工程与信息科学系，中国科学技术大学）；Department of Electrical and Computer Engineering, University of Delaware（电气与计算机工程系，德克萨斯大学）
ROSA-TFormer: A Radar-Optical Sensor-Aware Temporal Transformer for Pinus sylvestris Plantation Classification in Northern Shaanxi Using GEE-Derived Sentinel-1/2 Time Series
Key Laboratory of Remote Sensing and Digital Earth, Aerospace Information Research Institute, Chinese Academy of Sciences (AIRCAS)（遥感与数字地球重点实验室，航天信息研究所，中国科学院（AIRCAS））
