arXiv 2412.095962024-12-12cs.CVcs.AIcs.CL

InternLM-XComposer2.5-OmniLive：面向长期流式音视频交互的综合多模态系统

InternLM-XComposer2.5-OmniLive: A Comprehensive Multimodal System for Long-term Streaming Video and Audio Interactions

The Chinese University of Hong Kong（香港中文大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；SenseTime Group（商汤集团）

Pan Zhang, Xiaoyi Dong, Yuhang Cao, Yuhang Zang, Rui Qian, Xilin Wei, Lin Chen, Yifei Li, Junbo Niu, Shuangrui Ding, Qipeng Guo, Haodong Duan, Xin Chen, Han Lv,… 展开作者

Pan Zhang, Xiaoyi Dong, Yuhang Cao, Yuhang Zang, Rui Qian, Xilin Wei, Lin Chen, Yifei Li, Junbo Niu, Shuangrui Ding, Qipeng Guo, Haodong Duan, Xin Chen, Han Lv, Zheng Nie, Min Zhang, Bin Wang, Wenwei Zhang, Xinyue Zhang, Jiaye Ge, Wei Li, Jingwen Li, Zhongying Tu, Conghui He, Xingcheng Zhang, Kai Chen, Yu Qiao, Dahua Lin, Jiaqi Wang

备注：: Github Repo: https://github.com/InternLM/InternLM-XComposer/tree/main/InternLM-XComposer-2.5-OmniLive

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对现有多模态大语言模型难以同时进行流式感知、记忆与推理，且长上下文存储历史数据效率低的问题，研究提出InternLM-XComposer2.5-OmniLive框架，通过解耦的三类模块实现流式音视频实时交互，模拟类人认知以提供长期自适应服务。

摘要：

打造能像人类认知一样与环境进行长期交互的AI系统，是一项长期存在的研究目标。近年来，多模态大语言模型（MLLMs）在开放世界理解方面取得了显著进展，但连续、同步的流式感知、记忆与推理挑战仍未得到充分探索。当前的MLLMs受限于其序列到序列架构，无法同时处理输入和生成响应，类似于无法在感知的同时进行思考。此外，依赖长上下文存储历史数据对于长期交互而言并不实用，因为保留所有信息的成本高昂且效率低下。因此，本项目并未依赖单一基础模型执行所有功能，而是从“专才通才AI（Specialized Generalist AI）”概念中汲取灵感，引入了解耦的流式感知、推理与记忆机制，实现了与流式音视频输入的实时交互。所提出的框架InternLM-XComposer2.5-OmniLive（IXC2.5-OL）包含三个核心模块：（1）流式感知模块：实时处理多模态信息，将关键细节存入记忆，并根据用户查询触发推理。（2）多模态长时记忆模块：整合短时记忆与长时记忆，将短时记忆压缩为长时记忆，以实现高效检索并提升准确率。（3）推理模块：响应查询并执行推理任务，与感知模块和记忆模块协同工作。本项目模拟了类人认知，使多模态大语言模型能够随时间提供连续且自适应的服务。

英文摘要：

Creating AI systems that can interact with environments over long periods, similar to human cognition, has been a longstanding research goal. Recent advancements in multimodal large language models (MLLMs) have made significant strides in open-world understanding. However, the challenge of continuous and simultaneous streaming perception, memory, and reasoning remains largely unexplored. Current MLLMs are constrained by their sequence-to-sequence architecture, which limits their ability to process inputs and generate responses simultaneously, akin to being unable to think while perceiving. Furthermore, relying on long contexts to store historical data is impractical for long-term interactions, as retaining all information becomes costly and inefficient. Therefore, rather than relying on a single foundation model to perform all functions, this project draws inspiration from the concept of the Specialized Generalist AI and introduces disentangled streaming perception, reasoning, and memory mechanisms, enabling real-time interaction with streaming video and audio input. The proposed framework InternLM-XComposer2.5-OmniLive (IXC2.5-OL) consists of three key modules: (1) Streaming Perception Module: Processes multimodal information in real-time, storing key details in memory and triggering reasoning in response to user queries. (2) Multi-modal Long Memory Module: Integrates short-term and long-term memory, compressing short-term memories into long-term ones for efficient retrieval and improved accuracy. (3) Reasoning Module: Responds to queries and executes reasoning tasks, coordinating with the perception and memory modules. This project simulates human-like cognition, enabling multimodal large language models to provide continuous and adaptive service over time.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。