Recently, self-supervised learning of depth and ego-motion from thermal images shows strong robustness and reliability under challenging scenarios. However, the inherent thermal image properties such as weak contrast, blurry edges, and noise hinder to generate effective self-supervision from thermal images. Therefore, most research relies on additional self-supervision sources such as well-lit RGB images, generative models, and Lidar information. In this paper, we conduct an in-depth analysis of thermal image characteristics that degenerates self-supervision from thermal images. Based on the analysis, we propose an effective thermal image mapping method that significantly increases image information, such as overall structure, contrast, and details, while preserving temporal consistency. The proposed method shows outperformed depth and pose results than previous state-of-the-art networks without leveraging additional RGB guidance.
The dominant paradigm for feature learning in computer vision relies on training neural networks for the task of object recognition using millions of hand labelled images. Is it possible to learn useful features for a diverse set of visual tasks using any other form of supervision? In biology, living organisms developed the ability of visual perception for the purpose of moving and acting in the world. Drawing inspiration from this observation, in this work we investigate if the awareness of egomotion can be used as a supervisory signal for feature learning. As opposed to the knowledge of clas…
Human-as-Humanoid: Enabling Zero-Shot Humanoid Learning from Ego-Exo Human Videos with Human-Aligned Embodiments
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；DeepCybo；ZGCA；ZGCI；Harbin Institute of Technology（哈尔滨工业大学）；Huazhong University of Science and Technology（华中科技大学）；Beihang University（北京航空航天大学）
A thermal camera can robustly capture thermal radiation images under harsh light conditions such as night scenes, tunnels, and disaster scenarios. However, despite this advantage, neither depth nor ego-motion estimation research for the thermal camera have not been actively explored so far. In this paper, we propose a self-supervised learning method for depth and ego-motion estimation from thermal images. The proposed method exploits multi-spectral consistency that consists of temperature and photometric consistency loss. The temperature consistency loss provides a fundamental self-supervisory…
Many model-based Visual Odometry (VO) algorithms have been proposed in the past decade, often restricted to the type of camera optics, or the underlying motion manifold observed. We envision robots to be able to learn and perform these tasks, in a minimally supervised setting, as they gain more experience. To this end, we propose a fully trainable solution to visual ego-motion estimation for varied camera optics. We propose a visual ego-motion learning architecture that maps observed optical flow vectors to an ego-motion density estimate via a Mixture Density Network (MDN). By modeling the arc…
Toyota Research Institute（丰田研究院）；Toyota Technological Institute at Chicago（芝加哥丰田技术学院）
Self-supervised learning has emerged as a powerful tool for depth and ego-motion estimation, leading to state-of-the-art results on benchmark datasets. However, one significant limitation shared by current methods is the assumption of a known parametric camera model -- usually the standard pinhole geometry -- leading to failure when applied to imaging systems that deviate significantly from this assumption (e.g., catadioptric cameras or underwater imaging). In this work, we show that self-supervision can be used to learn accurate depth and ego-motion estimation without prior knowledge of the c…
Discriminately Treating Motion Components Evolves Joint Depth and Ego-Motion Learning
Shanghai Research Institute for Intelligent Autonomous Systems, Tongji University（上海智能自主系统研究院，同济大学）；College of Electronic & Information Engineering, Tongji University（电子与信息工程学院，同济大学）；Department of Control Science and Engineering, Zhejiang University（控制科学与工程学院，浙江大学）；National Key Laboratory of Human-Machine Hybrid Augmented Intelligence, the National Engineering Research Center for Visual Information and Applications, and the Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University（人机混合增强智能国家重点实验室、视觉信息与应用国家工程研究中心、人工智能与机器人研究所，西安交通大学）；College of Electronic & Information Engineering, the School of Computer Science and Technology, and the Key Laboratory of Embedded System and Service Computing (Ministry of Education), Tongji University（电子与信息工程学院、计算机科学与技术学院、教育部嵌入式系统与服务计算重点实验室，同济大学）；College of Electronic & Information Engineering, Shanghai Institute of Intelligent Science and Technology, Shanghai Research Institute for Intelligent Autonomous Systems, the State Key Laboratory of Autonomous Intelligent Unmanned Systems, the Frontiers Science Center for Intelligent Autonomous Systems (Ministry of Education), and Shanghai Key Laboratory of Intelligent Autonomous Systems, Tongji University（电子与信息工程学院、上海智能科学与技术研究院、上海智能自主系统研究院、自主智能无人系统国家重点实验室、智能自主系统前沿科学中心（教育部）、上海智能自主系统重点实验室，同济大学）
Event-Free Moving Object Segmentation from Moving Ego Vehicle
University of Burgundy（勃艮第大学）；University of Wurzburg（维尔茨堡大学）；INSAIT（保加利亚国家研究与技术创新研究所）；Université Rouen Normandie（鲁昂诺曼底大学）；INSA Rouen Normandie（鲁昂诺曼底国立应用科学学院）；Université Le Havre Normandie（勒阿弗尔诺曼底大学）；Normandie Université（诺曼底大学）；ETH Zurich（苏黎世联邦理工学院）；CVL（计算机视觉实验室）
Learning depth and ego-motion from unlabeled videos via self-supervision from epipolar projection can improve the robustness and accuracy of the 3D perception and localization of vision-based robots. However, the rigid projection computed by ego-motion cannot represent all scene points, such as points on moving objects, leading to false guidance in these regions. To address this problem, we propose an Attentional Separation-and-Aggregation Network (ASANet), which can learn to distinguish and extract the scene's static and dynamic characteristics via the attention mechanism. We further propose …
University of Edinburgh（爱丁堡大学）；Delft University of Technology（代尔夫特理工大学）
This work proposes a novel approach to 4D radar-based scene flow estimation via cross-modal learning. Our approach is motivated by the co-located sensing redundancy in modern autonomous vehicles. Such redundancy implicitly provides various forms of supervision cues to the radar scene flow estimation. Specifically, we introduce a multi-task model architecture for the identified cross-modal learning problem and propose loss functions to opportunistically engage scene flow estimation using multiple cross-modal constraints for effective model training. Extensive experiments show the state-of-the-a…
For ego-motion estimation, the feature representation of the scenes is crucial. Previous methods indicate that both the low-level and semantic feature-based methods can achieve promising results. Therefore, the incorporation of hierarchical feature representation may benefit from both methods. From this perspective, we propose a novel direct feature odometry framework, named DFO, for depth estimation and hierarchical feature representation learning from monocular videos. By exploiting the metric distance, our framework is able to learn the hierarchical feature representation without supervisio…
