arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.22897cs.AIcs.CLcs.CVcs.LGcs.MM
收藏

OmniGAIA:迈向原生全模态AI代理

OmniGAIA: Towards Native Omni-Modal AI Agents

Renmin University of China(中国人民大学);Xiaohongshu Inc.(小红书公司);Peking University(北京大学);Southeast University(东南大学);Tsinghua University(清华大学)

Xiaoxi Li, Wenxiang Jiao, Jiarui Jin, Haoxuan Li, Hao Wang, Shijian Wang, Guanting Dong, Jiajie Jin, Yinuo Wang, Yuan Lu, Ji-Rong Wen, Zhicheng Dou, Zhouchen Lin

AI总结:

提出OmniGAIA基准和OmniAtlas代理,通过全模态事件图和后见引导树探索策略,实现跨视频、音频和图像的深度推理与工具使用。

摘要:

人类智能自然地融合了全模态感知——涵盖视觉、音频和语言——与复杂推理和工具使用,以与世界互动。然而,当前的多模态大语言模型主要局限于双模态交互(例如,视觉-语言),缺乏通用AI助手所需的统一认知能力。为弥合这一差距,我们引入了OmniGAIA,一个全面的基准,旨在评估全模态代理在需要深度推理和跨视频、音频、图像模态的多轮工具执行任务上的表现。通过一种新颖的全模态事件图方法构建,OmniGAIA综合了源自真实数据的复杂多跳查询,这些查询需要跨模态推理和外部工具集成。此外,我们提出了OmniAtlas,一种在工具集成推理范式下的原生全模态基础代理,具有主动全模态感知。通过后见引导树探索策略和OmniDPO进行细粒度错误校正合成的轨迹进行训练,OmniAtlas有效增强了现有开源模型的工具使用能力。这项工作标志着向现实场景中下一代原生全模态AI助手迈出了一步。

英文摘要:

Human intelligence naturally intertwines omni-modal perception -- spanning vision, audio, and language -- with complex reasoning and tool usage to interact with the world. However, current multi-modal LLMs are primarily confined to bi-modal interactions (e.g., vision-language), lacking the unified cognitive capabilities required for general AI assistants. To bridge this gap, we introduce OmniGAIA, a comprehensive benchmark designed to evaluate omni-modal agents on tasks necessitating deep reasoning and multi-turn tool execution across video, audio, and image modalities. Constructed via a novel omni-modal event graph approach, OmniGAIA synthesizes complex, multi-hop queries derived from real-world data that require cross-modal reasoning and external tool integration. Furthermore, we propose OmniAtlas, a native omni-modal foundation agent under tool-integrated reasoning paradigm with active omni-modal perception. Trained on trajectories synthesized via a hindsight-guided tree exploration strategy and OmniDPO for fine-grained error correction, OmniAtlas effectively enhances the tool-use capabilities of existing open-source models. This work marks a step towards next-generation native omni-modal AI assistants for real-world scenarios.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑