The unified audio-visual understanding capability of omni-modal large language models (om-LLMs) comes at the high cost of processing tens of thousands of visual and audio tokens throughout the LLM. To reduce the cost, training-free methods for om-LLM token selection are being actively developed, operating either pre-LLM or inner-LLM. Recognizing that the non-textual token redundancy varies across stages and LLM layers, we propose in this paper a novel stage-adaptive token selection (SEATS) method that operates in both pre-LLM and inner-LLM stages. Given an overall token budget measured by the token retention rate (TRR), SEATS performs top-down token budget allocation in an input-dependent manner. Specifically, a larger TRR is used for pre-LLM token selection, whilst within the LLM, all tokens are retained in shallow layers, increasingly decreased TRRs are applied in middle layers, and the non-textual tokens are fully removed in late layers. Extensive experiments on three representative om-LLMs and six benchmarks verify the viability of the proposed method. By retaining only 10\% of visual and audio tokens, SEATS preserves 94.9\% of the full-token performance on Qwen2.5-Omni while achieving a 9.3x FLOPs reduction and a 4.9x prefill speedup.
Omni-Reward: Towards Generalist Omni-Modal Reward Modeling with Free-Form Preferences
School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所)
MorphoQuant: Modality-Aware Quantization for Omni-modal Large Language Models
institutetext: MorphoQuant: Modality-Aware Quantization for Omni-modal Large Language Models Yue Wu Changyuan Wang Zixuan Wang Shilin Ma Yansong Tang(机构文本:MorphoQuant:多模态大语言模型的模态感知量化 Yue Wu 王昌元 王梓轩 马世林 唐彦松)
OmniBench: Towards The Future of Universal Omni-Language Models
University of Manchester(曼彻斯特大学);Queen Mary University of London(伦敦大学玛丽女王学院);Hongkong University of Science and Technology(香港科学与技术大学);Nanjing University(南京大学);Dartmouth College(达特茅斯学院)
Learning Generalized and Flexible Trajectory Models from Omni-Semantic Supervision
Southern University of Science and Technology(南方科技大学);City University of Hong Kong(香港城市大学);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳))
FutureOmni: Evaluating Future Forecasting from Omni-Modal Context for Multimodal LLMs
Fudan University(复旦大学);Shanghai Innovation Institute(上海创新研究院);Harbin Institute of Technology, Shenzhen(哈尔滨工业大学深圳分校);National University of Singapore(新加坡国立大学)
Omnimodal Dataset Distillation via High-order Proxy Alignment
University of Science and Technology of China(中国科学技术大学);Xidian University(西安电子科技大学);National University of Singapore(新加坡国立大学);The University of Sydney(悉尼大学)
OmniScope: Modality-Decoupled Token Compression for Omnimodal Large Language Models
Media Analytics and Computing Lab, Xiamen University(厦门大学媒体分析与计算实验室);Institute of Artificial Intelligence, Xiamen University(厦门大学人工智能研究所);Alibaba Group(阿里巴巴集团)
OmniFocus: Query-Guided Modality-Balanced Token Compression for Omni-Modal Large Language Models
School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences(中国科学院大学先进交叉科学学院);Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences(中国科学院软件研究所中文信息处理实验室);University of Limerick(利默里克大学);CUHK, Shenzhen(香港中文大学(深圳))
Institute of Trustworthy Embodied AI, Fudan University(复旦大学可信具身人工智能研究所);Shanghai Collaborative Innovation Center of Intelligent Visual Computing(上海智能视觉计算协同创新中心);Bytedance Seed(字节跳动Seed);The University of Hong Kong(香港大学)
ReID5o: Achieving Omni Multi-modal Person Re-identification in a Single Model
National Key Laboratory of Multispectral Information Intelligent Processing Technology, School of Artificial Intelligence and Automation, Huazhong University of Science and Technology(多谱信息智能处理技术国家实验室,人工智能与自动化学院,华中科技大学);Shanghai AI Laboratory(上海人工智能实验室)