OmniSIFT: Modality-Asymmetric Token Compression for Efficient Omni-modal Large Language Models
New Laboratory of Pattern Recognition (NLPR), Institute of Automation, Chinese Academy of Sciences (CASIA)(模式识别新实验室(NLPR)、自动化研究所、中国科学院(CASIA));Nanjing University(南京大学);The Hong Kong University of Science(香港科学大学);Sichuan University(四川大学);Peking University(北京大学)
Yue Ding, Yiyan Ji, Jungang Li, Xuyang Liu, Xinlong Chen, Junfei Wu, Bozhou Li, Bohan Zeng, Yang Shi, Yushuo Guan, Yuanxing Zhang, Jiaheng Liu, Qiang Liu, Pengfei Wan, Liang Wang
Omni-modal Large Language Models (Omni-LLMs) have demonstrated strong capabilities in audio-video understanding tasks. However, their reliance on long multimodal token sequences leads to substantial computational overhead. Despite this challenge, token compression methods designed for Omni-LLMs remain limited. To bridge this gap, we propose OmniSIFT (Omni-modal Spatio-temporal Informed Fine-grained Token compression), a modality-asymmetric token compression framework tailored for Omni-LLMs. Specifically, OmniSIFT adopts a two-stage compression strategy: (i) a spatio-temporal video pruning module that removes video redundancy arising from both intra-frame structure and inter-frame overlap, and (ii) a vision-guided audio selection module that filters audio tokens. The entire framework is optimized end-to-end via a differentiable straight-through estimator. Extensive experiments on five representative benchmarks demonstrate the efficacy and robustness of OmniSIFT. Notably, for Qwen2.5-Omni-7B, OmniSIFT introduces only 4.85M parameters while maintaining lower latency than training-free baselines such as OmniZip. With merely 25% of the original token context, OmniSIFT consistently outperforms all compression baselines and even surpasses the performance of the full-token model on several tasks.
Omni-Safety under Cross-Modality Conflict: Vulnerabilities, Dynamics Mechanisms and Efficient Alignment
Nanyang Technological University(南洋理工大学);Beijing University of Posts and Telecommunications(北京邮电大学);Tsinghua University(清华大学);Fudan University(复旦大学);University of Science and Technology of China(中国科学技术大学);Renmin University of China(中国人民大学)
Stream-Omni: Simultaneous Multimodal Interactions with Large Language-Vision-Speech Model
Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences (ICT/CAS)(中国科学院智能信息处理重点实验室);Key Laboratory of AI Safety, Chinese Academy of Sciences(中国科学院人工智能安全重点实验室);University of Chinese Academy of Sciences, Beijing, China(中国科学院大学)
OmniHallu: Unified Hallucination Detection for Cross-Modal Comprehension and Generation in Multimodal Large Language Models
The University of Manchester(曼彻斯特大学);The University of Hong Kong(香港大学);Huazhong University of Science and Technology(华中科技大学);Shanghai Academy of Educational Sciences(上海市教育科学研究院);National University of Singapore(新加坡国立大学)
LongVALE: Vision-Audio-Language-Event Benchmark Towards Time-Aware Omni-Modal Perception of Long Videos
Southern University of Science and Technology(南方科技大学);University of Birmingham(伯明翰大学);University of Electronic Science and Technology of China(电子科技大学);The University of Hong Kong(香港大学);University of Manchester(曼彻斯特大学)
Omni-Reward: Towards Generalist Omni-Modal Reward Modeling with Free-Form Preferences
School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所)
ROMA: Real-time Omni-Multimodal Assistant with Interactive Streaming Understanding
CAS Key Laboratory of AI Safety, Institute of Computing Technology, CAS, Beijing, China(中国科学院人工智能安全重点实验室,计算技术研究所,中国科学院,北京,中国);University of Chinese Academy of Sciences, Beijing, China(中国科学院大学,北京,中国);Tsinghua University, Beijing, China(清华大学,北京,中国)
OmniBench: Towards The Future of Universal Omni-Language Models
University of Manchester(曼彻斯特大学);Queen Mary University of London(伦敦大学玛丽女王学院);Hongkong University of Science and Technology(香港科学与技术大学);Nanjing University(南京大学);Dartmouth College(达特茅斯学院)
AR-Omni: A Unified Autoregressive Model for Any-to-Any Generation
The Hong Kong Polytechnic University(香港理工大学);University of Science and Technology of China(中国科学技术大学);Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳))
VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Dataset
School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Nanjing University of Science and Technology(南京理工大学)