EventGPT: Event Stream Understanding with Multimodal Large Language Models
Xidian University(西安电子科技大学);Tsinghua University(清华大学);UCAS(中国科学院大学);Peking University(北京大学);Guangdong Laboratory of Artificial Intelligence and Digital Economy(SZ)(广东省人工智能与数字经济实验室(深圳))
Shaoyu Liu, Jianing Li, Guanghui Zhao, Yunjian Zhang, Xin Meng, Fei Richard Yu, Xiangyang Ji, Ming Li
Event cameras record visual information as asynchronous pixel change streams, excelling at scene perception under unsatisfactory lighting or high-dynamic conditions. Existing multimodal large language models (MLLMs) concentrate on natural RGB images, failing in scenarios where event data fits better. In this paper, we introduce EventGPT, the first MLLM for event stream understanding, to the best of our knowledge, marking a pioneering attempt to integrate large language models (LLMs) with event stream comprehension. To mitigate the huge domain gaps, we develop a three-stage optimization paradigm to gradually equip a pre-trained LLM with the capability of understanding event-based scenes. Our EventGPT comprises an event encoder, followed by a spatio-temporal aggregator, a linear projector, an event-language adapter, and an LLM. Firstly, RGB image-text pairs generated by GPT are leveraged to warm up the linear projector, referring to LLaVA, as the gap between natural image and language modalities is relatively smaller. Secondly, we construct a synthetic yet large dataset, N-ImageNet-Chat, consisting of event frames and corresponding texts to enable the use of the spatio-temporal aggregator and to train the event-language adapter, thereby aligning event features more closely with the language space. Finally, we gather an instruction dataset, Event-Chat, which contains extensive real-world data to fine-tune the entire model, further enhancing its generalization ability. We construct a comprehensive benchmark, and experiments show that EventGPT surpasses previous state-of-the-art MLLMs in generation quality, descriptive accuracy, and reasoning capability.
A Multimodal Transformer for Live Streaming Highlight Prediction
School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);KuaiShou Inc.(快手公司);CAIR, HK Institute of Science and Innovation, Chinese Academy of Sciences(中国科学院香港创新研究院人工智能与机器人中心)
SpeakingFaces: A Large-Scale Multimodal Dataset of Voice Commands with Visual and Thermal Video Streams
Institute of Smart Systems and Artificial Intelligence, Nazarbayev University(纳扎尔巴耶夫大学智能系统与人工智能研究所);School of Engineering and Digital Sciences, Nazarbayev University(纳扎尔巴耶夫大学工程与数字科学学院)
Leveraging Generic Foundation Models for Multimodal Surgical Data Analysis
Department of Biomedical Engineering, University of Basel, Allschwil, Switzerland(巴塞尔大学生物医学工程系);Clarunis – University Digestive Health Care Center Basel, Basel, Switzerland(Clarunis – 巴塞尔大学消化健康医疗中心)
EventFlash: Towards Efficient MLLMs for Event-Based Vision
Xidian University(西安电子科技大学);Tsinghua University(清华大学);Beijing Institute of Technology(北京理工大学);Guangdong Laboratory of Artificial Intelligence and Digital Economy(SZ)(广东人工智能与数字经济实验室(深圳))
Hyperbolic Multimodal Continual Learning: A Closest-Admissible Solution
The Chinese University of Hong Kong(香港中文大学);National University of Singapore(新加坡国立大学);Yale University(耶鲁大学);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州))
Multimodal Hate Detection Using Dual-Stream Graph Neural Networks
Multimodal Intelligence Lab, Department of Computer Science University of Exeter Exeter, UK(埃克塞特大学计算机科学系多模态智能实验室);School of Computer Science University of Birmingham Birmingham, UK(伯明翰大学计算机科学学院)
StreamArena: Toward Continuous, Interactive, and Long-Horizon Agentic Streaming Video Understanding
The Hong Kong University of Science and Technology(香港科技大学);Xiaohongshu Inc.(小红书公司);The University of Hong Kong(香港大学);The Chinese University of Hong Kong(香港中文大学)
A Novel Two Stream Decision Level Fusion of Vision and Inertial Sensors Data for Automatic Multimodal Human Activity Recognition System
Academy of Scientific and Innovative Research (AcSIR)(科学与创新研究院);CSIR-Central Electronics Engineering Research Institute (CEERI)(CSIR-中央电子工程研究所);Birla Institute of Technology and Science Pilani(毕拉理工学院皮拉尼分校);Edge Hill University(边山大学)
Probing Multimodal Fusion in the Brain: The Dominance of Audiovisual Streams in Naturalistic Encoding
Distributed and Intelligent Optimization Research Laboratory(分布式智能优化研究实验室);Electrical Engineering Department(电气工程系);Amirkabir University of Technology(阿米尔卡比尔技术大学)
ROMA: Real-time Omni-Multimodal Assistant with Interactive Streaming Understanding
CAS Key Laboratory of AI Safety, Institute of Computing Technology, CAS, Beijing, China(中国科学院人工智能安全重点实验室,计算技术研究所,中国科学院,北京,中国);University of Chinese Academy of Sciences, Beijing, China(中国科学院大学,北京,中国);Tsinghua University, Beijing, China(清华大学,北京,中国)
MotionVerse: A Unified Multimodal Framework for Motion Comprehension, Generation and Editing
Key Laboratory of Intelligent Information Processing, Institute of Computing Technology (ICT), Chinese Academy of Sciences (CAS)(智能信息处理重点实验室,计算技术研究所(ICT),中国科学院(CAS));University of the Chinese Academy of Sciences(中国科学院大学)
DashFusion: Dual-stream Alignment with Hierarchical Bottleneck Fusion for Multimodal Sentiment Analysis
School of Artificial Intelligence, Beijing University of Posts and Telecommunications(北京邮电大学人工智能学院);Beijing National Research Center for Information Science and Technology, Tsinghua University(清华大学信息科学与技术国家研究中心);Department of Automation, Tsinghua University(清华大学自动化系)
EventVL: Understand Event Streams via Multimodal Large Language Model
The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));KU Leuven(根特大学);École Polytechnique Fédérale de Lausanne(洛桑联邦理工学院);Carleton University(卡尔顿大学)
TTMFN: Two-stream Transformer-based Multimodal Fusion Network for Survival Prediction
Hangzhou Dianzi University(杭州电子科技大学);Hangzhou Institute of Advanced Technology(杭州先进技术研究院);Communication University of Zhejiang(浙江传媒学院);Shenzhen Research Institute of BigData(深圳大数据研究院);Misr Higher Institute for Commerce and Computers(密斯尔商业与计算机高等学院);Zhejiang University School of Medicine(浙江大学医学院);Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences(中国科学院深圳先进技术研究院)
Memory-Augmented Multimodal Large Language Models for Small Object Understanding in Streaming Aerial Videos
The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));University of Freiburg(弗莱堡大学);Hangzhou City University(杭州城市大学);XGRIDS(XGRIDS公司);Fudan University(复旦大学);Hong Kong Baptist University(香港浸会大学)
Stream-Omni: Simultaneous Multimodal Interactions with Large Language-Vision-Speech Model
Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences (ICT/CAS)(中国科学院智能信息处理重点实验室);Key Laboratory of AI Safety, Chinese Academy of Sciences(中国科学院人工智能安全重点实验室);University of Chinese Academy of Sciences, Beijing, China(中国科学院大学)