arXiv 2605.220122026-05-21cs.CLcs.CV

LatentOmni: 通过统一的音频-视觉潜在推理重新思考多模态理解

LatentOmni: Rethinking Omni-Modal Understanding via Unified Audio-Visual Latent Reasoning

School of AI, Shanghai Jiao Tong University（上海交通大学人工智能学院）；Kling Team, Kuaishou Technology（快手科技 Kling 团队）；Peking University（北京大学）；HKUST（香港科技大学）；CASIA（中国科学院自动化研究所）；Nanjing University（南京大学）；Renmin University of China（中国人民大学）；Tsinghua University（清华大学）

Yifan Dai, Zhenhua Wu, Bohan Zeng, Daili Hua, Jialing Liu, Bozhou Li, Yuran Wang, Chengzhuo Tong, Hao Liang, Xiaochen Ma, Junbo Niu, Tianyu Guo, Yang Shi, Yue D… 展开作者

Yifan Dai, Zhenhua Wu, Bohan Zeng, Daili Hua, Jialing Liu, Bozhou Li, Yuran Wang, Chengzhuo Tong, Hao Liang, Xiaochen Ma, Junbo Niu, Tianyu Guo, Yang Shi, Yue Ding, Yiyan Ji, Bingyin Mei, Yushuo Guan, Yuanxing Zhang, Pengfei Wan, Fangcheng Fu, Wentao Zhang

备注：: 21 pages, 15 figures

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出LatentOmni框架，通过统一的音频-视觉潜在空间进行多模态推理，利用特征级监督和Omni-Sync Position Embedding保持时间一致性，从而在多个音频-视觉推理基准测试中取得最佳性能。

摘要：

联合音频-视觉推理对于多模态理解至关重要，但当前的多模态大语言模型（MLLMs）在需要从两种模态中提取细粒度证据进行推理时仍存在困难。一个核心限制是显式的基于文本的推理链（CoT）将连续的音频-视觉信号压缩成离散的标记，削弱了时间定位并使中间推理偏向语言先验。我们主张统一的潜在空间是此类推理更好的媒介，因为它保留了密集的感知信息，同时仍能与自回归生成兼容。基于这一见解，我们提出了LatentOmni，一个跨模态推理框架，将文本推理与音频-视觉潜在状态交织在一起。LatentOmni引入了特征级监督，以对齐潜在推理状态与任务相关的感知特征，并使用Omni-Sync Position Embedding（OSPE）来保持潜在音频和视觉状态之间的时间一致性。我们进一步构建了LatentOmni-Instruct-35K数据集，该数据集包含音频-视觉交织推理轨迹，用于监督潜在空间推理。在多个音频-视觉推理基准测试中的综合评估表明，LatentOmni在评估的开源模型中取得了最佳性能，并且在显式文本CoT基线中表现一致，支持潜在空间联合推理作为更强多模态理解的有前途的路径。

英文摘要：

Joint audio-visual reasoning is essential for omnimodal understanding, yet current multimodal large language models (MLLMs) still struggle when reasoning requires fine-grained evidence from both modalities. A central limitation is that explicit text-based chain-of-thought (CoT) compresses continuous audio-visual signals into discrete tokens, weakening temporal grounding and shifting intermediate reasoning toward language priors. We argue that a unified latent space is a better medium for such reasoning because it preserves dense sensory information while remaining compatible with autoregressive generation. Based on this insight, we propose \textbf{LatentOmni}, a cross-modal reasoning framework that interleaves textual reasoning with audio-visual latent states. LatentOmni introduces feature-level supervision to align latent reasoning states with task-relevant sensory features and uses Omni-Sync Position Embedding (OSPE) to maintain temporal consistency between latent audio and visual states. We further construct \textbf{LatentOmni-Instruct-35K}, a dataset of audio-visual interleaved reasoning trajectories for supervising latent-space reasoning. Comprehensive evaluation across multiple audio-visual reasoning benchmarks demonstrates that LatentOmni achieves the best performance among the evaluated open-source models and consistently outperforms the Explicit Text CoT baseline, supporting latent-space joint reasoning as a promising path toward stronger omnimodal understanding.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。