MiniCPM-o 4.5：迈向实时全双工多模态交互
MiniCPM-o 4.5: Towards Real-Time Full-Duplex Omni-Modal Interaction
AI总结：
MiniCPM-o 4.5通过实时全双工多模态交互技术，解决多模态交互中感知与响应分离及被动响应的问题，实现更接近人类水平的多模态交互能力。
摘要：
近年来，多模态大语言模型（MLLMs）的进展使AI能力从静态离线数据处理扩展到实时流式交互，但仍未达到人类水平的多模态交互。关键瓶颈不再只是模态覆盖或延迟，而是交互范式本身。首先，感知与响应仍分为交替阶段，阻碍模型在生成过程中及时调整。其次，大多数现有模型仍处于被动响应状态，仅对明确用户请求作出反应，而非在演变的多模态环境中主动行动。我们提出MiniCPM-o 4.5，即我们向人类级多模态交互的最新努力，通过实时全双工多模态交互弥合这些差距。它能同时实时视觉、听觉和语音交互，并基于对实时场景的持续理解主动执行提醒或评论等行为。MiniCPM-o 4.5的核心技术是Omni-Flow，一种统一的流式框架，将多模态输入和输出沿共享时间轴对齐。这种表述将传统回合式交互转换为全双工、时间对齐的过程，使同时感知与响应成为可能，并在相同框架内产生主动行为。拥有90亿参数的MiniCPM-o 4.5在视觉-语言能力上接近Gemini 2.5 Flash，其规模在开源性能上达到最佳。它在多模态理解上超越Qwen3-Omni-30B-A3B，并在语音生成方面表现更优，计算效率显著更高。得益于其高效的架构设计和推理优化，该模型能够在RAM成本低于12GB的边缘设备上执行实时全双工多模态交互。
英文摘要：
Recent progress in multimodal large language models (MLLMs) has brought AI capabilities from static offline data processing to real-time streaming interaction, yet they still remain far from human-level multimodal interaction. The key bottlenecks are no longer modality coverage or latency alone, but the interaction paradigm itself. First, perception and response are still separated into alternating phases, preventing models from incorporating new inputs for timely adjustment during generation. Second, most current models remain reactive, responding only to explicit user requests instead of acting proactively in the evolving multimodal environment. We present MiniCPM-o 4.5, our latest effort towards human-like multimodal interaction, which mitigates these gaps by real-time full-duplex omni-modal interaction. It can see, listen, and speak simultaneously in real-time, while also exhibiting proactive behaviors such as issuing reminders or comments based on its continuous understanding of the live scene. The key technique behind MiniCPM-o 4.5 is Omni-Flow, a unified streaming framework that aligns omni-modal inputs and outputs along a shared temporal axis. This formulation converts conventional turn-based interaction into a full-duplex, time-aligned process, enabling simultaneous perception and response and allowing proactive behavior to arise within the same framework. With a total of 9B parameters, MiniCPM-o 4.5 approaches Gemini 2.5 Flash in vision-language capabilities, delivering state-of-the-art open-source performance at its scale. It also surpasses Qwen3-Omni-30B-A3B in omni-modal understanding and delivers better speech generation, with significantly higher computation efficiency. Driven by its efficient architecture design and inference optimization, the model can perform real-time full-duplex omni-modal interaction on edge devices with less than 12GB RAM cost.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
