In this article, we propose a novel LiDAR and event camera fusion modality for subterranean (SubT) environments for fast and precise object and human detection in a wide variety of adverse lighting conditions, such as low or no light, high-contrast zones and in the presence of blinding light sources. In the proposed approach, information from the event camera and LiDAR are fused to localize a human or an object-of-interest in a robot's local frame. The local detection is then transformed into the inertial frame and used to set references for a Nonlinear Model Predictive Controller (NMPC) for reactive tracking of humans or objects in SubT environments. The proposed novel fusion uses intensity filtering and K-means clustering on the LiDAR point cloud and frequency filtering and connectivity clustering on the events induced in an event camera by the returning LiDAR beams. The centroids of the clusters in the event camera and LiDAR streams are then paired to localize reflective markers present on safety vests and signs in SubT environments. The efficacy of the proposed scheme has been experimentally validated in a real SubT environment (a mine) with a Pioneer 3AT mobile robot. The experimental results show real-time performance for human detection and the NMPC-based controller allows for reactive tracking of a human or object of interest, even in complete darkness.
IIIT Hyderabad（印度海得拉巴国际信息技术学院）；ZF TCI（采埃孚技术中心印度）；Queensland University of Technology (QUT)（昆士兰科技大学）
Detecting objects under adverse weather and lighting conditions is crucial for the safe and continuous operation of an autonomous vehicle, and remains an unsolved problem. We present a Gated Differentiable Image Processing (GDIP) block, a domain-agnostic network architecture, which can be plugged into existing object detection networks (e.g., Yolo) and trained end-to-end with adverse condition images such as those captured under fog and low lighting. Our proposed GDIP block learns to enhance images directly through the downstream object detection loss. This is achieved by learning parameters o…
Developing, Analyzing, and Evaluating Vehicular Lane Keeping Algorithms Under Dynamic Lighting and Weather Conditions Using Electric Vehicles
Rice University（莱斯大学）；Pennsylvania State University（宾夕法尼亚州立大学）；Willamette University（威拉姆特大学）；Louisiana State University（路易斯安那州立大学）；Michigan State University（密歇根州立大学）；Lawrence Technological University（劳伦斯理工大学）
Adapting Depth Anything to Adverse Imaging Conditions with Events
National Key Lab of Multispectral Information Intelligent Processing Technology, School of Artificial Intelligence and Automation, Huazhong University of Science and Technology（multispectral information intelligent processing technology 国家重点实验室，人工智能与自动化学院，华中科技大学）；School of Computing, National University of Singapore（computing 学院，新加坡国立大学）；School of Computer Science and Engineering, Sun Yat-Sen University（computer science and engineering 学院，中山大学）
Deep Depth Estimation from Thermal Image: Dataset, Benchmark, and Challenges
School of Computer Science, Carnegie Mellon University（卡内基梅隆大学计算机科学学院）；School of Computer Science and Engineering, Pusan National University（釜山国立大学计算机科学与工程学院）
CompSLAM: Complementary Hierarchical Multi-Modal Localization and Mapping for Robot Autonomy in Underground Environments
Jet Propulsion Lab, California Institute of Technology, USA（喷气推进实验室，加州理工学院，美国）；Division of Robotics, Perception, and Learning, KTH Royal Institute of Technology, Sweden（机器人、感知与学习系，皇家理工学院，瑞典）；Autonomous Systems Lab, ETH Zürich, Switzerland（自主系统实验室，苏黎世联邦理工学院，瑞士）；Robotic Systems Lab, ETH Zürich, Switzerland（机器人系统实验室，苏黎世联邦理工学院，瑞士）；Autonomous Robots Lab, NTNU, Trondheim, Norway（自主机器人实验室，挪威特罗姆瑟大学，挪威）
Institute of Mathematical and Computer Sciences (ICMC)（数学与计算机科学学院（ICMC））
One of the main paths towards the reduction of traffic accidents is the increase in vehicle safety through driver assistance systems or even systems with a complete level of autonomy. In these types of systems, tasks such as obstacle detection and segmentation, especially the Deep Learning-based ones, play a fundamental role in scene understanding for correct and safe navigation. Besides that, the wide variety of sensors in vehicles nowadays provides a rich set of alternatives for improvement in the robustness of perception in challenging situations, such as navigation under lighting and weath…
University of Edinburgh（爱丁堡大学）；SAIC Motor Technical Center（上汽集团技术中心）
Scene flow allows autonomous vehicles to reason about the arbitrary motion of multiple independent objects which is the key to long-term mobile autonomy. While estimating the scene flow from LiDAR has progressed recently, it remains largely unknown how to estimate the scene flow from a 4-D radar - an increasingly popular automotive sensor for its robustness against adverse weather and lighting conditions. Compared with the LiDAR point clouds, radar data are drastically sparser, noisier and in much lower resolution. Annotated datasets for radar scene flow are also in absence and costly to acqui…
Photometric stereo provides an important method for high-fidelity 3D reconstruction based on multiple intensity images captured under different illumination directions. In this paper, we present a complete framework, including a multilight source illumination and acquisition hardware system and a two-stage convolutional neural network (CNN) architecture, to construct inter- and intraframe representations for accurate normal estimation of non-Lambertian objects. We experimentally investigate numerous network design alternatives for identifying the optimal scheme to deploy inter- and intraframe …
WaterScenes: A Multi-Task 4D Radar-Camera Fusion Dataset and Benchmarks for Autonomous Driving on Water Surfaces
University of Liverpool（利物浦大学）；Wuhan University of Technology（武汉理工大学）；Nantong University（南通大学）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））
Depth-aware Fusion Method based on Image and 4D Radar Spectrum for 3D Object Detection
The Global Institute of Future Technology, Shanghai Jiao Tong University（上海交通大学未来全球技术研究院）；Shanghai Jiao Tong University（上海交通大学）；Key Laboratory of System Control and Information Processing, Ministry of Education of China（教育部系统控制与信息处理重点实验室）
CubeDN: Real-time Drone Detection in 3D Space from Dual mmWave Radar Cubes
Department of Security and Crime Science, University College London（安全与犯罪科学系，伦敦大学学院）；Department of Computer Science, University College London（计算机科学系，伦敦大学学院）
Zhejiang University（浙江大学）；Hunan University（湖南大学）；College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；Shanghai SUPREMIND Technology Company Ltd.（上海思普睿明科技有限公司）
VLM-CASE: Vision-Language Model Enabled Context-Adaptive Safety Envelopes for Anticipatory Safe Autonomous Driving
Department of Civil and Environmental Engineering, The Pennsylvania State University（宾夕法尼亚州立大学土木与环境工程系）；Department of Civil Engineering, Stony Brook University（石溪大学土木工程系）
Visually-guided underwater robots are deployed alongside human divers for cooperative exploration, inspection, and monitoring tasks in numerous shallow-water and coastal-water applications. The most essential capability of such companion robots is to visually interpret their surroundings and assist the divers during various stages of an underwater mission. Despite recent technological advancements, the existing systems and solutions for real-time visual perception are greatly affected by marine artifacts such as poor visibility, lighting variation, and the scarcity of salient features. The dif…
Jet Propulsion Laboratory (JPL), California Institute of Technology (Caltech)（加州理工学院喷气推进实验室（JPL））；Swiss Federal Institute of Technology, Zurich（苏黎世联邦理工学院）；Massachusetts Institute of Technology（麻省理工学院）
We propose a framework for resilient autonomous navigation in perceptually challenging unknown environments with mobility-stressing elements such as uneven surfaces with rocks and boulders, steep slopes, negative obstacles like cliffs and holes, and narrow passages. Environments are GPS-denied and perceptually-degraded with variable lighting from dark to lit and obscurants (dust, fog, smoke). Lack of prior maps and degraded communication eliminates the possibility of prior or off-board computation or operator intervention. This necessitates real-time on-board computation using noisy sensor dat…
This letter presents a self-contained system for robust deployment of autonomous aerial vehicles in environments without access to global navigation systems and with limited lighting conditions. The proposed system, application-tailored for documentation in dark areas of large historical monuments, uses a unique and reliable aerial platform with a multi-modal lightweight sensory setup to acquire data in human-restricted areas with adverse lighting conditions, especially in areas that are high above the ground. The introduced localization method relies on an easy-to-obtain 3-D point cloud of a …
Robust Roadside Perception: an Automated Data Synthesis Pipeline Minimizing Human Annotation
University of Michigan（密歇根大学）；University of Michigan Transportation Research Institute（密歇根大学交通研究所）；Beihang University（北京航空航天大学）；Mcity（Mcity（密歇根大学智能城市实验室））
