发表机构
Adobe Research; University of California Davis（Adobe研究院; 加州大学戴维斯分校）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
OmniSeek将全模态大模型转变为主动多轮推理智能体，通过动态决定查看或聆听的时间窗口来检索关键证据，并利用合成轨迹与两阶段强化学习优化，显著提升跨模态音视频推理性能。
AI 中文摘要
我们提出了OmniSeek，一个智能体框架，它将全模态大语言模型（Omni-LLM）转变为一个具备原生工具使用的主动多轮推理智能体。OmniSeek并非在单次前向传播中被动地处理整个音视频序列，而是将证据获取作为推理过程的一部分：它动态决定是查看还是聆听，以及针对哪个时间窗口，以在长上下文中跨不同模态检索稀疏但关键的证据。通过迭代的多轮协议，检索到的原始音频或视觉片段被追加回上下文中，以支持后续推理。为了冷启动这一能力，我们构建了一个数据引擎，合成了OmniTraj-170K，这是一个包含交错音频和视觉证据的多跳思维链轨迹语料库。我们首先在这些轨迹上监督模型，以灌输多轮工具使用行为，然后通过两阶段可验证奖励的强化学习进一步优化策略。此外，我们引入了一个音视频必要性目标，该目标明确奖励那些推理依赖于两种模态的成功轨迹，从而抑制单模态捷径。在广泛基准上的大量实验表明，OmniSeek学会了自适应的跨模态证据搜索，并持续提升了音视频推理性能。
英文摘要
We present OmniSeek, an agentic framework that transforms an Omni Large Language Model (Omni-LLM) into an active, multi-turn reasoning agent with native tool use. Rather than passively processing an entire audio-visual sequence in a single forward pass, OmniSeek makes evidence acquisition part of the reasoning process: it dynamically decides whether to look or listen, and over which temporal window, to retrieve sparse but critical evidence across different modalities within long contexts. Through an iterative multi-turn protocol, the retrieved raw audio or visual segments are appended back into the context to support subsequent reasoning. To cold-start this capability, we build a data engine that synthesizes OmniTraj-170K, a corpus of multi-hop Chain-of-Thought trajectories with interleaved audio and visual evidence. We first supervise the model on these trajectories to instill multi-turn tool-use behavior, and then further optimize the policy via a two-stage reinforcement learning with verifiable rewards. Moreover, we introduce an Audio-Visual Necessity objective that explicitly rewards successful trajectories whose reasoning depends on both modalities, discouraging single-modality shortcuts. Extensive experiments across a wide range of benchmarks demonstrate that OmniSeek learns adaptive cross-modal evidence seeking and consistently improves audio-visual reasoning performance.
