Event cameras provide several unique advantages over standard frame-based sensors, including high temporal resolution, low latency, and robustness to extreme lighting. However, existing learning-based approaches for event processing are typically confined to narrow, task-specific silos and lack the ability to generalize across modalities. We address this gap with REALM, a cross-modal framework that learns an RGB- and Event-Aligned Latent Manifold by projecting event representations into the pretrained latent space of RGB foundation models. Instead of task-specific training, we leverage low-rank adaptation (LoRA) to bridge the modality gap, effectively unlocking the geometric and semantic priors of frozen RGB backbones for asynchronous event streams. We demonstrate that REALM effectively maps events into the ViT-based foundation latent space. Our method performs downstream tasks, such as depth estimation and semantic segmentation, by simply transferring linear heads trained on the RGB teacher. Most significantly, REALM enables the direct, zero-shot application of complex, frozen image-trained decoders, such as MASt3R, to raw event data. We demonstrate state-of-the-art performance in wide-baseline feature matching, significantly outperforming specialized architectures. Code and models are available at https://papers.starslab.ca/realm/.
Explicit Attention-Enhanced Fusion for RGB-Thermal Perception Tasks
Shenzhen Institute of Artificial Intelligence and Robotics for Society(深圳人工智能与机器人研究院);School of Science and Engineering, The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳)理工学院)
Unveiling the Potential of Segment Anything Model 2 for RGB-Thermal Semantic Segmentation with Language Guidance
School of Artificial Intelligence and Robotics, Hunan University(人工智能与机器人学院,湖南大学);National Engineering Research Center of Robot Visual Perception and Control Technology, Hunan University(机器人视觉感知与控制技术国家工程研究中心,湖南大学);AI Thrust, Hong Kong University of Science and Technology (Guangzhou)(香港理工大学(广州)人工智能方向)
InfraNet: Quality-Aware RGB Guidance for Efficient Infrared Object Detection
Beihang University(北京航空航天大学);Zhongguancun Academy(中关村科学城);Communication University of China(中国传媒大学);King Abdullah University of Science and Technology(阿卜杜拉国王科技大学)
Event-RGB Adaptive Tracking for Nighttime Highway Perception
School of Intelligent Systems Engineering, Sun Yat-sen University(中山大学智能系统工程学院);School of Intelligence Science and Engineering, Harbin Institute of Technology(哈尔滨工业大学智能科学与工程学院)
Bridging the RGB-IR Gap: Consensus and Discrepancy Modeling for Text-Guided Multispectral Detection
Department of Automation, Tsinghua University(清华大学自动化系);State Key Laboratory of Blockchain and Data Security, Zhejiang University(浙江大学区块链与数据安全全国重点实验室);Tsinghua University(清华大学)
TUNI: Unifying Pre-training and Fine-tuning with Modality-Aware Mutual Learning and Rectification for RGB-T Semantic Segmentation
School of Automation, Beijing Institute of Technology(自动化学院,北京理工大学);School of Computer Science, Wuhan University(计算机学院,武汉大学);Great Wall Motor(长城汽车);School of Information and Electronic Engineering, Zhejiang University of Science and Technology(信息电子工程学院,浙江理工大学)
Modality Dominance-Aware Optimization for Embodied RGB-Infrared Perception
College of Electronics and Information Engineering(电子信息工程学院);School of Computer Science and Technology(计算机科学与技术学院);Department of Electrical and Computer Engineering(电气与计算机工程系);School of Future Science and Engineering(未来科学与工程学院);Key Laboratory of General Artificial Intelligence and Large Models in Provincial Universities(省属大学通用人工智能与大模型重点实验室)
JRN-Geo: A Joint Perception Network based on RGB and Normal images for Cross-view Geo-localization
College of Information Science and Engineering, Northeastern University(信息科学与工程学院,东北大学);School of Computer Science, University of Sheffield(计算机科学学院,谢菲尔德大学)
Towards Privacy-Preserving Thermal Human Perception from Dataset to Deployment
Massachusetts Institute of Technology(麻省理工学院);Singapore-MIT Alliance for Research and Technology Centre (SMART)(新加坡-麻省理工研究与技术联盟中心(SMART));Chongqing General Hospital(重庆总医院);Chongqing University(重庆大学);Hong Kong Xsmart Century Technology Co., Ltd.(香港智享世纪科技有限公司);National University of Singapore(新加坡国立大学);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));Tsinghua University(清华大学)
MMPhysVideo: Physically Plausible Video Generation Through Joint RGB-Perception Modeling
State Key Laboratory of Multimodal Artificial Intelligence Systems, CASIA(中国科学院自动化研究所多模态人工智能系统国家重点实验室);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);StepFun(阶跃星辰);Beijing Key Laboratory of Super Intelligent Security of Multi-Modal Information(多模态信息超级智能安全北京市重点实验室);School of Information Science and Technology, Shanghai Tech University(上海科技大学信息科学与技术学院)