arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2412.00832cs.CV
收藏

EventGPT:基于多模态大语言模型的事件流理解

EventGPT: Event Stream Understanding with Multimodal Large Language Models

Xidian University(西安电子科技大学);Tsinghua University(清华大学);UCAS(中国科学院大学);Peking University(北京大学);Guangdong Laboratory of Artificial Intelligence and Digital Economy(SZ)(广东省人工智能与数字经济实验室(深圳))

Shaoyu Liu, Jianing Li, Guanghui Zhao, Yunjian Zhang, Xin Meng, Fei Richard Yu, Xiangyang Ji, Ming Li

AI总结:

提出首个面向事件流理解的多模态大语言模型EventGPT,通过事件编码器、时空聚合器与事件-语言适配器及三阶段训练,在综合基准上超越现有MLLMs。

摘要:

事件相机将视觉信息记录为异步像素变化流,擅长在光照不理想或高动态条件下进行场景感知。现有多模态大语言模型(MLLMs)主要关注自然RGB图像,在更适合事件数据的场景中表现不佳。本文提出EventGPT,据我们所知,这是首个用于事件流理解的MLLM,标志着将大语言模型(LLMs)与事件流理解相结合的开创性尝试。为缓解巨大的领域差距,我们开发了一种三阶段优化范式,使预训练LLM逐步具备理解基于事件的场景的能力。我们的EventGPT由事件编码器、时空聚合器、线性投影器、事件-语言适配器和LLM组成。首先,参考LLaVA,利用GPT生成的RGB图像-文本对来预热线性投影器,因为自然图像与语言模态之间的差距相对较小。其次,我们构建了一个大型合成数据集N-ImageNet-Chat,包含事件帧和相应文本,以启用时空聚合器并训练事件-语言适配器,从而使事件特征与语言空间更紧密地对齐。最后,我们收集了包含大量真实世界数据的指令数据集Event-Chat,对整个模型进行微调,进一步增强其泛化能力。我们构建了一个全面的基准测试,实验表明EventGPT在生成质量、描述准确性和推理能力方面均超越了以往最先进的MLLMs。

英文摘要:

Event cameras record visual information as asynchronous pixel change streams, excelling at scene perception under unsatisfactory lighting or high-dynamic conditions. Existing multimodal large language models (MLLMs) concentrate on natural RGB images, failing in scenarios where event data fits better. In this paper, we introduce EventGPT, the first MLLM for event stream understanding, to the best of our knowledge, marking a pioneering attempt to integrate large language models (LLMs) with event stream comprehension. To mitigate the huge domain gaps, we develop a three-stage optimization paradigm to gradually equip a pre-trained LLM with the capability of understanding event-based scenes. Our EventGPT comprises an event encoder, followed by a spatio-temporal aggregator, a linear projector, an event-language adapter, and an LLM. Firstly, RGB image-text pairs generated by GPT are leveraged to warm up the linear projector, referring to LLaVA, as the gap between natural image and language modalities is relatively smaller. Secondly, we construct a synthetic yet large dataset, N-ImageNet-Chat, consisting of event frames and corresponding texts to enable the use of the spatio-temporal aggregator and to train the event-language adapter, thereby aligning event features more closely with the language space. Finally, we gather an instruction dataset, Event-Chat, which contains extensive real-world data to fine-tune the entire model, further enhancing its generalization ability. We construct a comprehensive benchmark, and experiments show that EventGPT surpasses previous state-of-the-art MLLMs in generation quality, descriptive accuracy, and reasoning capability.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑