Human intelligence naturally intertwines omni-modal perception -- spanning vision, audio, and language -- with complex reasoning and tool usage to interact with the world. However, current multi-modal LLMs are primarily confined to bi-modal interactions (e.g., vision-language), lacking the unified cognitive capabilities required for general AI assistants. To bridge this gap, we introduce OmniGAIA, a comprehensive benchmark designed to evaluate omni-modal agents on tasks necessitating deep reasoning and multi-turn tool execution across video, audio, and image modalities. Constructed via a novel omni-modal event graph approach, OmniGAIA synthesizes complex, multi-hop queries derived from real-world data that require cross-modal reasoning and external tool integration. Furthermore, we propose OmniAtlas, a native omni-modal foundation agent under tool-integrated reasoning paradigm with active omni-modal perception. Trained on trajectories synthesized via a hindsight-guided tree exploration strategy and OmniDPO for fine-grained error correction, OmniAtlas effectively enhances the tool-use capabilities of existing open-source models. This work marks a step towards next-generation native omni-modal AI assistants for real-world scenarios.
Omni-Decision: Evidence-Ledger Planning for Omni-Modal Agents
Institute of Neuroscience, Chinese Academy of Sciences(中国科学院神经科学研究所);University of Chinese Academy of Sciences(中国科学院大学);Tongyi Lab, Alibaba Group(阿里巴巴集团通义实验室);Tsinghua University(清华大学);Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所)
NExT-OMNI: Towards Any-to-Any Omnimodal Foundation Models with Discrete Flow Matching
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences(中国科学院深圳先进技术研究院);University of Chinese Academy of Sciences(中国科学院大学);NExT++ Research Center(NExT++研究中心);National University of Singapore(新加坡国立大学)
OmniVChat: Synthesizing, Benchmarking, and Training for Native Audio-Visual Dialogue
The Chinese University of Hong Kong(香港中文大学);Alibaba Token Hub, Alibaba Group(阿里巴巴集团通义实验室);Shanghai Jiao Tong University(上海交通大学);Shanghai Innovation Institute(上海创新研究院);Zhejiang University(浙江大学)
BayLing-Duplex: Native Full-Duplex Speech Dialogue with a Single Autoregressive LLM
Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences (ICT/CAS)(中国科学院计算技术研究所智能信息处理重点实验室);Key Laboratory of AI Safety, Chinese Academy of Sciences(中国科学院人工智能安全重点实验室);University of Chinese Academy of Sciences(中国科学院大学)
v-HUB: A Benchmark for Video Humor Understanding from Vision and Sound
Shanghai Jiao Tong University(上海交通大学);Wuhan University(武汉大学);Beijing Institute for General Artificial Intelligence(北京一般人工智能研究院);Independent Researcher(独立研究者)
Beyond Text-Dominance: Understanding Modality Preference of Omni-modal Large Language Models
University of Chinese Academy of Sciences(中国科学院大学);Chinese Information Processing Laboratory(中文信息处理实验室);Institute of Software, Chinese Academy of Sciences(中国科学院软件研究所)
OmniTide: Co-Designing Algorithms and Systems for Efficient On-Device Omni-LLM Streaming
Key Lab of High Confidence Software Technologies (Peking University)(高可信软件技术教育部重点实验室(北京大学));State Key Laboratory of Networking and Switching Technology (BUPT)(网络与交换技术国家重点实验室(北京邮电大学))