Recent learning-based approaches have achieved impressive results in the field of single-shot camera localization. However, how best to fuse multiple modalities (e.g., image and depth) and to deal with degraded or missing input are less well studied. In particular, we note that previous approaches towards deep fusion do not perform significantly better than models employing a single modality. We conjecture that this is because of the naive approaches to feature space fusion through summation or concatenation which do not take into account the different strengths of each modality. To address this, we propose an end-to-end framework, termed VMLoc, to fuse different sensor inputs into a common latent space through a variational Product-of-Experts (PoE) followed by attention-based fusion. Unlike previous multimodal variational works directly adapting the objective function of vanilla variational auto-encoder, we show how camera localization can be accurately estimated through an unbiased objective function based on importance weighting. Our model is extensively evaluated on RGB-D datasets and the results prove the efficacy of our model. The source code is available at https://github.com/kaichen-z/VMLoc.
This paper describes a multimodal vision sensor that integrates three types of cameras, including a stereo camera, a polarization camera and a panoramic camera. Each sensor provides a specific dimension of information: the stereo camera measures depth per pixel, the polarization obtains the degree of polarization, and the panoramic camera captures a 360-degree landscape. Data fusion and advanced environment perception could be built upon the combination of sensors. Designed especially for autonomous driving, this vision sensor is shipped with a robust semantic segmentation network. In addition…
Thinking with Camera: A Unified Multimodal Model for Camera-Centric Understanding and Generation
S-Lab, Nanyang Technological University（南洋理工大学S-Lab）；SenseTime Research（商汤科技研究院）；University of Michigan（密歇根大学）；Max-Planck Institute for Informatics（马克斯·普朗克信息学研究所）
Technical University of Munich（慕尼黑工业大学）；Stanford University（斯坦福大学）；Siemens AG（西门子股份公司）；ETH Zurich（苏黎世联邦理工学院）；Johns Hopkins University（约翰斯·霍普金斯大学）
We present a multimodal camera relocalization framework that captures ambiguities and uncertainties with continuous mixture models defined on the manifold of camera poses. In highly ambiguous environments, which can easily arise due to symmetries and repetitive structures in the scene, computing one plausible solution (what most state-of-the-art methods currently regress) may not be sufficient. Instead we predict multiple camera pose hypotheses as well as the respective uncertainty for each prediction. Towards this aim, we use Bingham distributions, to model the orientation of the camera pose,…
University of Liège（列日大学）；Aalborg University（奥尔堡大学）
Monitoring the occupancy of public sports facilities is essential to assess their use and to motivate their construction in new places. In the case of a football field, the area to cover is large, thus several regular cameras should be used, which makes the setup expensive and complex. As an alternative, we developed a system that detects players from a unique cheap and wide-angle fisheye camera assisted by a single narrow-angle thermal camera. In this work, we train a network in a knowledge distillation approach in which the student and the teacher have different modalities and a different vi…
OmniCam: Unified Multimodal Video Generation via Camera Control
Zhejiang University（浙江大学）；Shanghai Jiao Tong University（上海交通大学）；Beijing University of Technology（北京工业大学）；Jiangnan University（江南大学）；University of Science and Technology of China（中国科学技术大学）；Nanjing university（南京大学）
Multimodal Biometric Authentication Using Camera-Based PPG and Fingerprint Fusion
King Abdullah University of Science and Technology（阿卜杜拉国王科技大学）；University of Toronto（多伦多大学）；King Fahad University of Petroleum and Minerals（法赫德国王石油与矿业大学）
In this paper, we design a multimodal framework for object detection, recognition and mapping based on the fusion of stereo camera frames, point cloud Velodyne Lidar scans, and Vehicle-to-Vehicle (V2V) Basic Safety Messages (BSMs) exchanged using Dedicated Short Range Communication (DSRC). We merge the key features of rich texture descriptions of objects from 2D images, depth and distance between objects provided by 3D point cloud and awareness of hidden vehicles from BSMs' 3D information. We present a joint pixel to point cloud and pixel to V2V correspondences of objects in frames from the Ki…
AlignMiF: Geometry-Aligned Multimodal Implicit Field for LiDAR-Camera Joint Synthesis
Shenzhen Campus of Sun Yat-sen University（中山大学深圳校区）；DarkMatter AI Research（DarkMatter AI研究）；University of Oxford（牛津大学）；HKU（香港大学）；Cainiao Group（菜鸟集团）；NCUT（北方工业大学）；Sun Yat-sen University（中山大学）；Westlake University（西湖大学）
University of Southern California（南加州大学）；U.S. Army Research Laboratory（美国陆军研究实验室）
Current perception systems often carry multimodal imagers and sensors such as 2D cameras and 3D LiDAR sensors. To fuse and utilize the data for downstream perception tasks, robust and accurate calibration of the multimodal sensor data is essential. We propose a novel deep learning-driven technique (CalibDNN) for accurate calibration among multimodal sensor, specifically LiDAR-Camera pairs. The key innovation of the proposed work is that it does not require any specific calibration targets or hardware assistants, and the entire processing is fully automatic with a single model and single iterat…
Human action recognition has been widely used in many fields of life, and many human action datasets have been published at the same time. However, most of the multi-modal databases have some shortcomings in the layout and number of sensors, which cannot fully represent the action features. Regarding the problems, this paper proposes a freely available dataset, named CZU-MHAD (Changzhou University: a comprehensive multi-modal human action dataset). It consists of 22 actions and three modals temporal synchronized data. These modals include depth videos and skeleton positions from a kinect v2 ca…
Multimodal Foundation Models for Zero-shot Animal Species Recognition in Camera Trap Images
Microsoft AI for Good Lab, Redmond（微软人工智能公益实验室）；University of Southern California, Los Angeles（南加州大学）；Microsoft Research, Redmond（微软研究院）；Nanyang Technological University, Singapore（南洋理工大学）；Universidad de Los Andes, Colombia（洛斯安第斯大学）
Dense Road Surface Grip Map Prediction from Multimodal Image Data
Finnish Geospatial Research Institute FGI, National Land Survey of Finland（芬兰地理空间研究所（FGI），芬兰国家土地测量局）；Department of Computer Science, Aalto University（阿尔托大学计算机科学系）
In this work, a deep learning approach has been developed to carry out road detection by fusing LIDAR point clouds and camera images. An unstructured and sparse point cloud is first projected onto the camera image plane and then upsampled to obtain a set of dense 2D images encoding spatial information. Several fully convolutional neural networks (FCNs) are then trained to carry out road detection, either by using data from a single sensor, or by using three fusion strategies: early, late, and the newly proposed cross fusion. Whereas in the former two fusion approaches, the integration of multi…
Utilizing the sensor characteristics of the audio, visible camera, and thermal camera, the robustness of person recognition can be enhanced. Existing multimodal person recognition frameworks are primarily formulated assuming that multimodal data is always available. In this paper, we propose a novel trimodal sensor fusion framework using the audio, visible, and thermal camera, which addresses the missing modality problem. In the framework, a novel deep latent embedding framework, termed the AVTNet, is proposed to learn multiple latent embeddings. Also, a novel loss function, termed missing mod…
A Generative AI Integrated Multimodal Framework for Low-Latency Multi-Camera Person Re-Identification
University of Moratuwa（莫拉图瓦大学）；Zone24x7 (Pvt) Ltd（Zone24x7（私人）有限公司）；Chulalongkorn University（朱拉隆功大学）；University of Colombo（科伦坡大学）；La Trobe University（拉筹伯大学）
VideoWeaver: Multimodal Multi-View Video-to-Video Transfer for Embodied Agents
Huawei Heisenberg Research Center（华为海森堡研究中心）；Ludwig Maximilian University of Munich（慕尼黑大学）；University of Freiburg（弗莱堡大学）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）
High-accuracy per-pixel depth is vital for computational photography, so smartphones now have multimodal camera systems with time-of-flight (ToF) depth sensors and multiple color cameras. However, producing accurate high-resolution depth is still challenging due to the low resolution and limited active illumination power of ToF sensors. Fusing RGB stereo and ToF information is a promising direction to overcome these issues, but a key problem remains: to provide high-quality 2D RGB images, the main color sensor's lens is optically stabilized, resulting in an unknown pose for the floating lens t…
SCAFusion: A Multimodal 3D Detection Framework for Small Object Detection in Lunar Surface Exploration
Department of Automation, Key Laboratory of System Control and Information Processing of Ministry of Education, Key Laboratory of Marine Intelligent Equipment and System of Ministry of Education, Shanghai Engineering Research Center of Intelligent Control and Management, Shanghai Jiao Tong University（自动化系、教育部系统控制与信息处理重点实验室、教育部海洋智能装备与系统重点实验室、上海智能控制与管理工程研究中心、上海交通大学）
Robotics and Perception Group, University of Zurich（苏黎世大学机器人与感知组）
Event cameras are novel vision sensors that report per-pixel brightness changes as a stream of asynchronous "events". They offer significant advantages compared to standard cameras due to their high temporal resolution, high dynamic range and lack of motion blur. However, events only measure the varying component of the visual signal, which limits their ability to encode scene context. By contrast, standard cameras measure absolute intensity frames, which capture a much richer representation of the scene. Both sensors are thus complementary. However, due to the asynchronous nature of events, c…
University of Science, Ho Chi Minh City（胡志明市科学大学）；Vietnam National University in Ho Chi Minh City（胡志明市越南国家大学）；AISIA Research Lab（AISIA研究实验室）；Hong Bang International University（鸿邦国际大学）
In recent years, the occurrence of falls has increased and has had detrimental effects on older adults. Therefore, various machine learning approaches and datasets have been introduced to construct an efficient fall detection algorithm for the social community. This paper studies the fall detection problem based on a large public dataset, namely the UP-Fall Detection Dataset. This dataset was collected from a dozen of volunteers using different sensors and two cameras. We propose several techniques to obtain valuable features from these sensors and cameras and then construct suitable models fo…
We present the Multiview Extended Video with Activities (MEVA) dataset, a new and very-large-scale dataset for human activity recognition. Existing security datasets either focus on activity counts by aggregating public video disseminated due to its content, which typically excludes same-scene background video, or they achieve persistence by observing public areas and thus cannot control for activity content. Our dataset is over 9300 hours of untrimmed, continuous video, scripted to include diverse, simultaneous activities, along with spontaneous background activity. We have annotated 144 hour…
The fusion of multimodal sensor streams, such as camera, lidar, and radar measurements, plays a critical role in object detection for autonomous vehicles, which base their decision making on these inputs. While existing methods exploit redundant information in good environmental conditions, they fail in adverse weather where the sensory streams can be asymmetrically distorted. These rare "edge-case" scenarios are not represented in available datasets, and existing fusion architectures are not designed to handle them. To address this challenge we present a novel multimodal dataset acquired in o…
Nanyang Technological University（南洋理工大学）；University of California, Berkeley（加州大学伯克利分校）；University of Edinburgh（爱丁堡大学）
As an important biomarker for human identification, human gait can be collected at a distance by passive sensors without subject cooperation, which plays an essential role in crime prevention, security detection and other human identification applications. At present, most research works are based on cameras and computer vision techniques to perform gait recognition. However, vision-based methods are not reliable when confronting poor illuminations, leading to degrading performances. In this paper, we propose a novel multimodal gait recognition method, namely GaitFi, which leverages WiFi signa…
