This report accompanies a dataset release on visual and thermal camera data and details a procedure followed to align such multi-modal camera frames in order to provide pixel-level correspondence between the two without using intrinsic or extrinsic calibration information. To achieve this goal we benefit from progress in the domain of multi-modal image alignment and specifically employ the Mattes Mutual Information Metric to guide the registration process. In the released dataset we release both the raw visual and thermal camera data, as well as the aligned frames, alongside calibration parameters with the goal to better facilitate the investigation on common local/global features across such multi-modal image streams.
CalFuse: Multi-Modal Continual Learning via Feature Calibration and Parameter Fusion
College of Intelligent Robotics and Advanced Manufacturing, Fudan University（智能机器人与先进制造学院，复旦大学）；School of Future Science and Engineering, Soochow University（未来科学与工程学院，苏州大学）；DataLab: Data Science and Informatics, University of California, Davis（数据实验室：数据科学与信息学，加州大学戴维斯分校）；University of Rochester（罗切斯特大学）；Ningbo University（宁波大学）；Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Beijing University of Posts and Telecommunications（北京邮电大学）；Academy for Computer Science and Informatics, Cardiff University（计算机科学与信息学学院，卡迪夫大学）
Perceive and Calibrate: Analyzing and Enhancing Robustness of Medical Multi-Modal Large Language Models
Department of Computer Science and Engineering, CUHK, Hong Kong, China（计算机科学与工程系，CUHK，香港，中国）；Institute of Medical Intelligence and XR, CUHK, Hong Kong, China（医学智能与XR研究所，CUHK，香港，中国）
DST-Calib: A Dual-Path, Self-Supervised, Target-Free LiDAR-Camera Extrinsic Calibration Network
Department of Control Science & Engineering, the College of Electronics & Information Engineering, Tongji University（控制科学与工程系，电子与信息工程学院，同济大学）；Department of Mechanical Engineering, the School of Mechanical Engineering, Tongji University（机械工程系，机械工程学院，同济大学）；School of Robotics, Hunan University（机器人学院，湖南大学）；College of Intelligence Science and Technology, National University of Defense Technology（智能科学与技术学院，国防科学技术大学）；Department of Control Science & Engineering, the College of Electronics & Information Engineering, Shanghai Research Institute for Intelligent Autonomous Systems, the State Key Laboratory of Intelligent Autonomous Systems, and Frontiers Science Center for Intelligent Autonomous Systems, Tongji University（控制科学与工程系，电子与信息工程学院，上海智能自主系统研究院，智能自主系统国家重点实验室，智能自主系统前沿科学中心，同济大学）；National Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University（人机混合增强智能国家重点实验室，人工智能与机器人研究院，西安交通大学）
The development process of high-fidelity SLAM systems depends on their validation upon reliable datasets. Towards this goal, we propose IBISCape, a simulated benchmark that includes data synchronization and acquisition APIs for telemetry from heterogeneous sensors: stereo-RGB/DVS, Depth, IMU, and GPS, along with the ground truth scene segmentation and vehicle ego-motion. Our benchmark is built upon the CARLA simulator, whose back-end is the Unreal Engine rendering a high dynamic scenery simulating the real world. Moreover, we offer 34 multi-modal datasets suitable for autonomous vehicles navig…
In this work we propose long wave infrared (LWIR) imagery as a viable supporting modality for semantic segmentation using learning-based techniques. We first address the problem of RGB-thermal camera calibration by proposing a passive calibration target and procedure that is both portable and easy to use. Second, we present PST900, a dataset of 894 synchronized and calibrated RGB and Thermal image pairs with per pixel human annotations across four distinct classes from the DARPA Subterranean Challenge. Lastly, we propose a CNN architecture for fast semantic segmentation that combines both RGB …
Nanyang Technological University（南洋理工大学）；Institute for Infocomm Research, A*STAR（新加坡科技研究局资讯通信研究院）
Assuming the source label space subsumes the target one, Partial Video Domain Adaptation (PVDA) is a more general and practical scenario for cross-domain video classification problems. The key challenge of PVDA is to mitigate the negative transfer caused by the source-only outlier classes. To tackle this challenge, a crucial step is to aggregate target predictions to assign class weights by up-weighing target classes and down-weighing outlier classes. However, the incorrect predictions of class weights can mislead the network and lead to negative transfer. Previous works improve the class weig…
CLRNet: Targetless Extrinsic Calibration for Camera, Lidar and 4D Radar Using Deep Learning
Hungarian Research Network Institute for Computer Science and Control (HUN-REN SZTAKI)（匈牙利研究网络计算机科学与控制研究所）；Pázmány Péter Catholic University（帕兹曼·佩斯特天主教大学）；Intelligent Vehicles Group, TU Delft（智能车辆组，代尔夫特理工大学）；Perciv AI
With information from multiple input modalities, sensor fusion-based algorithms usually out-perform their single-modality counterparts in robotics. Camera and LIDAR, with complementary semantic and depth information, are the typical choices for detection tasks in complicated driving environments. For most camera-LIDAR fusion algorithms, however, the calibration of the sensor suite will greatly impact the performance. More specifically, the detection algorithm usually requires an accurate geometric relationship among multiple sensors as the input, and it is often assumed that the contents from …
Georgia Institute of Technology（佐治亚理工学院）；Wuhan University of Technology（武汉理工大学）
This paper introduces a two-phase deep feature calibration framework for efficient learning of semantics enhanced text-image cross-modal joint embedding, which clearly separates the deep feature calibration in data preprocessing from training the joint embedding model. We use the Recipe1M dataset for the technical description and empirical validation. In preprocessing, we perform deep feature calibration by combining deep feature engineering with semantic context features derived from raw text-image input data. We leverage LSTM to identify key terms, NLP methods to produce ranking scores for k…
Multi-modal MRI Translation via Evidential Regression and Distribution Calibration
Fudan University（复旦大学）；University of Cambridge（剑桥大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Fuzhou University（福州市大学）；Shanghai Jiao Tong University（上海交通大学）
UniCalib: Targetless LiDAR-Camera Calibration via Probabilistic Flow on Unified Depth Representations
School of Computer Science, Wuhan University（武汉大学计算机学院）；School of Electronic Information, Wuhan University（武汉大学电子信息学院）；School of Artificial Intelligence, Wuhan University（武汉大学人工智能学院）
Cross-Modal Attention Calibration for LVLM Hallucination Mitigation
Sun Yat-sen University（中山大学）；The University of Hong Kong（香港大学）；Meituan（美团）；Inspur Database Technology（Inspur数据库技术）；Guilin University of Electronic Technology（桂林电子科技大学）；Shenzhen Loop Area Institute（深圳河套学院）；Guangdong Key Laboratory of Big Data Analysis and Processing（广东大数据分析与处理重点实验室）
FoCal: Frequency-Oriented Cross-Modal Interaction and Spectral Calibration for Aerial Visible-Infrared Object Detection
Nanjing University of Science and Technology（南京理工大学）；Shanghai Institute of Technical Physics, Chinese Academy of Sciences（中国科学院上海技术物理研究所）；China Electronics Technology Group Corporation（中国电子科技集团公司）
Online,Target-Free LiDAR-Camera Extrinsic Calibration via Cross-Modal Mask Matching
Tongji University（同济大学）；College of Electronics & Information Engineering（电子与信息工程学院）；Shanghai Institute of Intelligent Science and Technology（上海智能科学技术研究院）；Shanghai Research Institute for Intelligent Autonomous Systems（上海智能自主系统研究院）；State Key Laboratory of Intelligent Autonomous Systems（智能自主系统国家重点实验室）；Frontiers Science Center for Intelligent Autonomous Systems（智能自主系统前沿科学中心）
Dynamic Alignment and Calibration for Multimodal Learning
School of Computer Science and Engineering, University of Electronic Science and Technology of China（电子科技大学计算机科学与工程学院）；Tianyijiaotong Technology Ltd.（天翼交通科技有限公司）；School of Computer Science and Technology, Hainan University（海南大学计算机科学与技术学院）
Native-Domain Cross-Attention for Camera-LiDAR Extrinsic Calibration Under Large Initial Perturbations
School of Automation, Beijing Institute of Technology（北京理工大学自动化学院）；School of Integrated Circuits and Electronics, Beijing Institute of Technology（北京理工大学集成电路与电子学院）
School of Mathematics and Statistics, Changsha University of Science and Technology（数学与统计学学院，长沙理工大学）；School of Biomedical Engineering, Shanghai Jiao Tong University（生物医学工程学院，上海交通大学）；Shanghai Chest Hospital, Shanghai Jiao Tong University School of Medicine（上海胸科医院，上海交通大学医学院）
CalibNet: Dual-branch Cross-modal Calibration for RGB-D Salient Instance Segmentation
School of Computer Science and Engineering, The Chinese University of Hong Kong（香港中文大学计算机科学与工程学院）；School of Software Engineering, Huazhong University of Science and Technology（华中科技大学软件学院）；Mohamed Bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；National University of Singapore（新加坡国立大学）；Nankai International Advanced Research Institute (Shenzhen Futian), Nankai University（南开大学南开大学南开国际高等研究院（深圳福田））；College of Computer Science, Nankai University（南开大学计算机学院）
Accurate sensor calibration is a prerequisite for multi-sensor perception and localization systems for autonomous vehicles. The intrinsic parameter calibration of the sensor is to obtain the mapping relationship inside the sensor, and the extrinsic parameter calibration is to transform two or more sensors into a unified spatial coordinate system. Most sensors need to be calibrated after installation to ensure the accuracy of sensor measurements. To this end, we present OpenCalib, a calibration toolbox that contains a rich set of various sensor calibration methods. OpenCalib covers manual calib…
4D-CAAL: 4D Radar-Camera Calibration and Auto-Labeling for Autonomous Driving
School of Information Engineering, Yancheng Institute of Technology（信息工程学院，盐城职业技术学院）；School of Navigation, Wuhan University of Technology（导航学院，武汉理工大学）；The Hong Kong University of Science and Technology (Guangzhou)（香港科学与技术大学（广州））；School of Information Engineering, Yancheng Institute Technology（信息工程学院，盐城职业技术学院）；School of Advanced Technology, Xi’an Jiaotong-Liverpool University（先进技术学院，西安交通大学利物浦大学）；School of Information Science and Technology, Nantong University（信息科学与技术学院，南通大学）；State Key Laboratory of Maritime Technology and Safety（船舶技术与安全国家重点实验室）
Tactile perception is an essential ability of intelligent robots in interaction with their surrounding environments. This perception as an intermediate level acts between sensation and action and has to be defined properly to generate suitable action in response to sensed data. In this paper, we propose a feedback approach to address robot grasping task using force-torque tactile sensing. While visual perception is an essential part for gross reaching, constant utilization of this sensing modality can negatively affect the grasping process with overwhelming computation. In such case, human bei…
Department of Control Science & Engineering, the College of Electronic & Information Engineering, Tongji University（控制科学与工程系，电子与信息工程学院，同济大学）；School of Computer Science and Engineering, Central South University（计算机科学与工程学院，中南大学）；Beijing Institute of Aerospace Control Devices（北京航天控制器件研究所）；Institute of Automation, Chinese Academy of Sciences（自动化研究所，中国科学院）
BOOKAGENT: Orchestrating Safety-Aware Visual Narratives via Multi-Agent Cognitive Calibration
Carnegie Mellon University（卡内基梅隆大学）；University of Science and Technology of China（中国科学技术大学）；Imperial College London（伦敦帝国理工学院）；Nanyang Technological University（南洋理工大学）；University of Central Florida（佛罗里达中央大学）
This paper presents a framework for the targetless extrinsic calibration of stereo cameras and Light Detection and Ranging (LiDAR) sensors with a non-overlapping Field of View (FOV). In order to solve the extrinsic calibrations problem under such challenging configuration, the proposed solution exploits road markings as static and robust features among the various dynamic objects that are present in urban environment. First, this study utilizes road markings that are commonly captured by the two sensor modalities to select informative images for estimating the extrinsic parameters. In order to…
Few-Shot Learning with Visual Distribution Calibration and Cross-Modal Distribution Alignment
Beihang University（北京航空航天大学）；Huawei Noah’s Ark Lab（华为诺亚方舟实验室）；Tokyo Institute of Technology（东京工业大学）；University of Science and Technology of China（中国科学技术大学）；Zhongguancun Laboratory（中关村实验室）
calibfusion: Transformer-Based Differentiable Calibration for Radar-Camera Fusion Detection in Water-Surface Environments
School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）
In many visual systems, visual tracking often bases on RGB image sequences, in which some targets are invalid in low-light conditions, and tracking performance is thus affected significantly. Introducing other modalities such as depth and infrared data is an effective way to handle imaging limitations of individual sources, but multi-modal imaging platforms usually require elaborate designs and cannot be applied in many real-world applications at present. Near-infrared (NIR) imaging becomes an essential part of many surveillance cameras, whose imaging is switchable between RGB and NIR based on…
CHRep: Cross-modal Histology Representation and Post-hoc Calibration for Spatial Gene Expression Prediction
Beijing University of Posts and Telecommunications（北京邮电大学）；Beijing Key Laboratory of Network System and Network Culture（北京网络系统与网络文化重点实验室）；Peking University Third Hospital（北京大学第三医院）
SCMM: Calibrating Cross-modal Representations for Text-Based Person Search
College of Future Information Technology, Fudan University（复旦大学未来信息技术学院）；Department of Electrical and Computer Engineering, The University of British Columbia（不列颠哥伦比亚大学电气与计算机工程系）；College of Electronic and Information Engineering, Tongji University（同济大学电子与信息工程学院）；MEGVII Technology（MEGVII技术）；School of Computer Science and Informatics, Cardiff University（卡迪夫大学计算机科学与信息学院）；School of AI and CS, Nantong University（南通大学人工智能与计算机科学学院）；Academy of Artificial Intelligence, SMBU（SMBU人工智能学院）；College of Computer Science and Software Engineering, Shenzhen University（深圳大学计算机科学与软件工程学院）
