多模态音频视频理解的主动感知代理
Active Perception Agent for Omnimodal Audio-Video Understanding
AI总结：
OmniAgent通过动态规划和音频引导感知范式，实现多模态细粒度推理，无需训练即超越现有模型性能。
摘要：
多模态大语言模型在统一音频和视觉模态方面取得了显著进展；然而，它们在细粒度跨模态理解和多模态对齐方面常常面临挑战。为了解决这些限制，我们引入了OmniAgent，据我们所知，这是第一个完全主动感知代理，动态协调专门的单模态工具以实现更细粒度的多模态推理。与以往依赖于刚性、静态工作流程和密集帧描述的工作不同，我们展示了一种从被动响应生成到主动多模态探究的范式转变。OmniAgent采用动态规划，自主地根据需求调用工具，战略性地将感知注意力集中在与任务相关的线索上。我们方法的核心是一种新的粗到细的音频引导感知范式，该范式利用音频线索来定位时间事件并指导后续推理。在三个音频视频理解基准上的广泛实证评估表明，OmniAgent实现了最先进的性能，无需训练即可在准确率上超越领先的开源和闭源模型，差距达10%至20%。
英文摘要：
Omnimodal large language models have made significant strides in unifying audio and visual modalities; however, they often face challenges in fine-grained cross-modal understanding and have difficulty with multimodal alignment. To address these limitations, we introduce OmniAgent, to our best knowledge, the first fully active perception agent that dynamically orchestrates specialized unimodal tools to achieve more fine-grained omnimodal reasoning. Unlike previous works that rely on rigid, static workflows and dense frame-captioning, we demonstrate a paradigm shift from passive response generation to active multimodal inquiry. OmniAgent employs dynamic planning to autonomously orchestrate tool invocation on demand, strategically concentrating perceptual attention on task-relevant cues. Central to our approach is a novel coarse-to-fine audio-guided perception paradigm, which leverages audio cues to localize temporal events and guide subsequent reasoning. Extensive empirical evaluations on three audio-video understanding benchmarks demonstrate that OmniAgent achieves state-of-the-art performance, surpassing leading open-source and closed-source models by substantial margins of 10% - 20% accuracy without training.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
