arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2605.20183cs.CV
收藏

MSAVBench:面向多镜头音频-视频生成的全面可靠评估

MSAVBench: Towards Comprehensive and Reliable Evaluation of Multi-Shot Audio-Video Generation

Fudan University(复旦大学);The University of Hong Kong(香港大学);Tongyi Lab, Alibaba Group(阿里集团通义实验室);Zhejiang University(浙江大学);Peking University(北京大学)
Yujie Wei, Yujin Han, Zhekai Chen, Yongming Li, Kaixun Jiang, Zhihang Liu, Quanhao Li, Zhiwu Qing, Xiang Wang, Zhen Xing, Ruihang Chu, Lingyi Hong, Yefei He, Ju… 展开作者

Yujie Wei, Yujin Han, Zhekai Chen, Yongming Li, Kaixun Jiang, Zhihang Liu, Quanhao Li, Zhiwu Qing, Xiang Wang, Zhen Xing, Ruihang Chu, Lingyi Hong, Yefei He, Junjie Zhou, Junqiu Yu, Yang Shi, Difan Zou, Kai Zhu, Shiwei Zhang, Yingya Zhang, Yu Liu, Xihui Liu, Hongming Shan

AI总结:

提出首个多镜头音频-视频生成基准MSAVBench,通过自适应混合评估框架在四个维度上系统评估19个模型,发现当前系统在导演级控制和细粒度音视频同步上仍存在挑战。

摘要:

视频生成正从单镜头合成快速演变为复杂的多镜头音频-视频(MSAV)叙事以满足现实需求。然而,评估此类前沿模型仍是一个基本挑战。现有基准在范围和数据多样性上有限,并依赖僵化的评估流程,阻碍了对现代MSAV模型的系统可靠评估。为弥补这些差距,我们引入MSAVBench,这是首个针对多镜头音频-视频生成的综合基准和自适应混合评估框架。我们的基准涵盖四个关键维度:视频、音频、镜头和参考,覆盖多样化的任务设置、多达15个镜头的可变数量以及具有挑战性的非真实场景。我们的评估框架通过镜头分割的自适应自校正机制、主观指标的实例化评分规则以及复杂判断的基于工具的证据提取,提高了鲁棒性。此外,MSAVBench与人类判断高度一致,达到91.5%的斯皮尔曼等级相关系数。我们对19个最先进的闭源和开源模型的系统评估表明,当前系统在导演级控制和细粒度音视频同步上仍存在困难,而模块化或代理式生成管道为缩小开源与闭源模型之间的差距提供了一条有希望的路径。基准数据和评估代码已在https://github.com/ali-vilab/MSAVBench公开。

英文摘要:

Video generation is rapidly evolving from single-shot synthesis to complex multi-shot audio-video (MSAV) narratives to meet real-world demands. However, evaluating such frontier models remains a fundamental challenge. Existing benchmarks are limited in scope and data diversity, and rely on rigid evaluation pipelines, preventing systematic and reliable assessment of modern MSAV models. To bridge these gaps, we introduce MSAVBench, the first comprehensive benchmark and adaptive hybrid evaluation framework for multi-shot audio-video generation. Our benchmark spans four key dimensions, video, audio, shot, and reference, covering diverse task settings, varying shot counts of up to 15, and challenging non-realistic scenarios. Our evaluation framework improves robustness through an adaptive self-correction mechanism for shot segmentation, instance-wise rubrics for subjective metrics, and tool-grounded evidence extraction for complex judgments. Furthermore, MSAVBench achieves high alignment with human judgments, reaching a Spearman rank correlation of 91.5%. Our systematic evaluation of 19 state-of-the-art closed- and open-source models shows that current systems still struggle with director-level control and fine-grained audio-visual synchronization, while modular or agentic generation pipelines offer a promising path toward narrowing the gap between open- and closed-source models. The benchmark data and evaluation code are publicly available at https://github.com/ali-vilab/MSAVBench.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑