arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2601.06573cs.AIcs.MM
收藏

QMAVIS:利用大多模态模型融合实现长视频音频理解

QMAVIS: Long Video-Audio Understanding using Fusion of Large Multimodal Models

National University of Singapore(新加坡国立大学);Nanyang Technological University, Singapore(南洋理工大学)

Zixing Lin, Jiale Wang, Gee Wah Ng, Lee Onn Mak, Chan Zhi Yang Jeriel, Jun Yang Lee, Yaohao Li

AI总结:

QMAVIS通过融合大型多模态模型、大型语言模型和语音识别模型,实现了长视频音频理解,展示了在VideoMME数据集上38.75%的性能提升,并在其他数据集上表现出竞争力。

摘要:

传统的大型多模态模型(LMMs)在视频音频理解方面通常仅在几分钟长的短视频上进行评估。本文介绍了QMAVIS(Q Team-Multimodal Audio Video Intelligent Sensemaking),一种通过后期融合LMMs、大型语言模型和语音识别模型构建的新型长视频音频理解流程。QMAVIS填补了长视频分析的空白,特别是针对几分钟到超过一小时长的视频,为感知、视频内容分析、具身AI等开辟了新的应用前景。使用QMAVIS进行定量实验表明,在包含音频信息的VideoMME数据集上,其性能比最先进的视频音频LMMs如VideoLlaMA2和InternVL2提高了38.75%。在其他具有挑战性的视频理解数据集如PerceptionTest和EgoSchema上,最高达到了2%的提升,显示出竞争力。定性实验也显示,QMAVIS能够提取长视频音频内容中不同场景的细微差别并理解整体叙事。此外,还进行了消融研究以确定融合流程中每个组件的影响。

英文摘要:

Large Multimodal Models (LMMs) for video-audio understanding have traditionally been evaluated only on shorter videos of a few minutes long. In this paper, we introduce QMAVIS (Q Team-Multimodal Audio Video Intelligent Sensemaking), a novel long video-audio understanding pipeline built through a late fusion of LMMs, Large Language Models, and speech recognition models. QMAVIS addresses the gap in long-form video analytics, particularly for longer videos of a few minutes to beyond an hour long, opening up new potential applications in sensemaking, video content analysis, embodied AI, etc. Quantitative experiments using QMAVIS demonstrated a 38.75% improvement over state-of-the-art video-audio LMMs like VideoLlaMA2 and InternVL2 on the VideoMME (with subtitles) dataset, which comprises long videos with audio information. Evaluations on other challenging video understanding datasets like PerceptionTest and EgoSchema saw up to 2% improvement, indicating competitive performance. Qualitative experiments also showed that QMAVIS is able to extract the nuances of different scenes in a long video audio content while understanding the overarching narrative. Ablation studies were also conducted to ascertain the impact of each component in the fusion pipeline.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑