RE-VLM：事件增强的视觉-语言模型用于场景理解
RE-VLM: Event-Augmented Vision-Language Model for Scene Understanding
AI总结：
本文提出RE-VLM，一种结合RGB图像和事件流的双流视觉-语言模型，旨在提升在正常和恶劣条件下对场景的理解能力。通过事件相机提供的高时间分辨率和宽动态范围的数据，RE-VLM在场景描述和视觉问答任务中优于现有模型。
摘要：
传统视觉-语言模型（VLMs）在恶劣条件下（如低光、高动态范围或快速运动）捕获的场景解释能力不足，因为标准RGB图像在这些环境中质量下降。事件相机提供了一种互补的模态：它们异步记录每个像素的亮度变化，具有高时间分辨率和宽动态范围，在帧失效时保留运动线索。我们提出了RE-VLM，第一个双流视觉-语言模型，联合利用RGB图像和事件流，以在正常和挑战性条件下实现稳健的场景理解。RE-VLM采用并行的RGB和事件编码器，以及一种渐进训练策略，将异构视觉特征与语言对齐。为了解决RGB-Event-Text监督不足的问题，我们进一步提出了一种图驱动的流程，将同步的RGB-Event流转换为可验证的场景图，从中合成描述和问答对。为了开发和评估RE-VLM，我们构建了两个数据集：PEOD-Chat，针对光照挑战性场景，和RGBE-Chat，涵盖多样化的场景。在描述和VQA基准测试中，RE-VLM在与现有RGB-only和事件-only模型参数量相当的情况下，始终优于现有模型，特别是在挑战性条件下表现显著提升。这些结果证明了事件增强的VLMs在广泛现实环境中实现稳健视觉-语言理解的有效性。
英文摘要：
Conventional vision-language models (VLMs) struggle to interpret scenes captured under adverse conditions (e.g., low light, high dynamic range, or fast motion) because standard RGB images degrade in such environments. Event cameras provide a complementary modality: they asynchronously record per-pixel brightness changes with high temporal resolution and wide dynamic range, preserving motion cues where frames fail. We propose RE-VLM, the first dual-stream vision-language model that jointly leverages RGB images and event streams for robust scene understanding across both normal and challenging conditions. RE-VLM employs parallel RGB and event encoders together with a progressive training strategy that aligns heterogeneous visual features with language. To address the scarcity of RGB-Event-Text supervision, we further propose a graph-driven pipeline that converts synchronized RGB-Event streams into verifiable scene graphs, from which we synthesize captions and question-answer (QA) pairs. To develop and evaluate RE-VLM, we construct two datasets: PEOD-Chat, targeting illumination-challenged scenes, and RGBE-Chat, covering diverse scenarios. On captioning and VQA benchmarks, RE-VLM consistently outperforms state-of-the-art RGB-only and event-only models with comparable parameter counts, with particularly large gains under challenging conditions. These results demonstrate the effectiveness of event-augmented VLMs in achieving robust vision-language understanding across a wide range of real-world environments.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
