Omni-Captioner：多模态信息细粒度感知的数据管道、模型与基准
Omni-Captioner: Data Pipeline, Models, and Benchmark for Omni Detailed Perception
AI总结：
本文提出Omni-Captioner，通过数据管道、模型和基准系统，系统研究多模态细粒度感知，提出Omni-Detective生成高质量数据，实现音频和视频细粒度描述的最优性能。
摘要：
多模态信息细粒度感知对于推进人机交互至关重要。随着音频视频技术的进步，能够并行处理音频和视频信号的Omni语言模型（OLMs）已成为实现更丰富理解和推理的有希望范式。然而，其捕捉和描述细粒度细节的能力仍有限。本文从数据管道、模型和基准的角度系统研究了多模态细粒度感知。我们首先发现当前OLMs中细节与幻觉之间存在固有的“共生”关系。为解决这一问题，我们提出了Omni-Detective，一种集成了工具调用的代理数据生成管道，以自动生成高细节但最小幻觉的多模态数据。基于Omni-Detective生成的数据，我们训练了两个描述模型：Audio-Captioner用于仅音频的细粒度感知，Omni-Captioner用于音频-视频细粒度感知。在级联评估协议下，Audio-Captioner在MMAU和MMAR中表现最佳，优于Gemini 2.5 Flash，并与Gemini 2.5 Pro性能相当。在现有细粒度描述基准上，Omni-Captioner在VDC上达到新状态，且在视频-SALMONN 2测试集上实现了细节与幻觉的最佳平衡。鉴于缺乏专门的多模态细粒度感知基准，我们设计了Omni-Cloze，一种新的cloze式评估，确保稳定、高效和可靠的评估。实验结果和分析证明了Omni-Detective在生成高质量细粒度描述中的有效性，以及Omni-Cloze在评估此类描述中的优越性。
英文摘要：
Fine-grained perception of multimodal information is critical for advancing human-AI interaction. With recent progress in audio-visual technologies, Omni Language Models (OLMs), capable of processing audio and video signals in parallel, have emerged as a promising paradigm for achieving richer understanding and reasoning. However, their capacity to capture and describe fine-grained details remains limited explored. In this work, we present a systematic and comprehensive investigation of omni detailed perception from the perspectives of the data pipeline, models, and benchmark. We first identify an inherent "co-growth" between detail and hallucination in current OLMs. To address this, we propose Omni-Detective, an agentic data generation pipeline integrating tool-calling, to autonomously produce highly detailed yet minimally hallucinatory multimodal data. Based on the data generated with Omni-Detective, we train two captioning models: Audio-Captioner for audio-only detailed perception, and Omni-Captioner for audio-visual detailed perception. Under the cascade evaluation protocol, Audio-Captioner achieves the best performance on MMAU and MMAR among all open-source models, surpassing Gemini 2.5 Flash and delivering performance comparable to Gemini 2.5 Pro. On existing detailed captioning benchmarks, Omni-Captioner sets a new state-of-the-art on VDC and achieves the best trade-off between detail and hallucination on the video-SALMONN 2 testset. Given the absence of a dedicated benchmark for omni detailed perception, we design Omni-Cloze, a novel cloze-style evaluation for detailed audio, visual, and audio-visual captioning that ensures stable, efficient, and reliable assessment. Experimental results and analysis demonstrate the effectiveness of Omni-Detective in generating high-quality detailed captions, as well as the superiority of Omni-Cloze in evaluating such detailed captions.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
