arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2605.00271cs.CVcs.AIcs.RO
收藏

REALM:一种RGB和事件对齐的潜在流形用于跨模态感知

REALM: An RGB- and Event-Aligned Latent Manifold for Cross-Modal Perception

University of Toronto, Robotics Institute(多伦多大学机器人研究所);University of Toronto, Department of Computer Science(多伦多大学计算机科学系)

Vincenzo Polizzi, David B. Lindell, Jonathan Kelly

AI总结:

REALM通过将事件表示投影到预训练的RGB基础模型潜在空间,实现跨模态感知,利用LoRA技术解锁冻结RGB主干的几何和语义先验,实现零样本应用复杂解码器。

摘要:

事件相机相比标准帧基传感器具有高时间分辨率、低延迟和对极端光照的鲁棒性。然而,现有基于学习的方法通常局限于狭窄的任务特定领域,缺乏跨模态泛化能力。我们通过REALM跨模态框架解决这一问题,该框架通过将事件表示投影到预训练的RGB基础模型潜在空间,学习RGB和事件对齐的潜在流形。我们利用低秩适应(LoRA)技术弥合模态差距,有效解锁冻结RGB主干的几何和语义先验,以处理异步事件流。我们证明REALM能有效将事件映射到基于ViT的基础潜在空间。我们的方法允许通过简单地转移在RGB教师上训练的线性头进行下游任务如深度估计和语义分割。最显著的是,REALM使复杂、冻结的图像训练解码器(如MASt3R)能够直接、零样本地应用于原始事件数据。我们在宽基线特征匹配中实现了最先进的性能,显著优于专用架构。代码和模型将在接受后提供。

英文摘要:

Event cameras provide several unique advantages over standard frame-based sensors, including high temporal resolution, low latency, and robustness to extreme lighting. However, existing learning-based approaches for event processing are typically confined to narrow, task-specific silos and lack the ability to generalize across modalities. We address this gap with REALM, a cross-modal framework that learns an RGB- and Event-Aligned Latent Manifold by projecting event representations into the pretrained latent space of RGB foundation models. Instead of task-specific training, we leverage low-rank adaptation (LoRA) to bridge the modality gap, effectively unlocking the geometric and semantic priors of frozen RGB backbones for asynchronous event streams. We demonstrate that REALM effectively maps events into the ViT-based foundation latent space. Our method performs downstream tasks, such as depth estimation and semantic segmentation, by simply transferring linear heads trained on the RGB teacher. Most significantly, REALM enables the direct, zero-shot application of complex, frozen image-trained decoders, such as MASt3R, to raw event data. We demonstrate state-of-the-art performance in wide-baseline feature matching, significantly outperforming specialized architectures. Code and models are available at https://papers.starslab.ca/realm/.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑