arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2401.07781cs.CV
收藏

迈向更优的文本到视频生成评估指标

Towards A Better Metric for Text-to-Video Generation

ARC Lab, Tencent PCG(腾讯PCG ARC实验室);Show Lab;Nanyang Technological University(南洋理工大学);National University of Singapore(新加坡国立大学)
Jay Zhangjie Wu, Guian Fang, Haoning Wu, Xintao Wang, Yixiao Ge, Xiaodong Cun, David Junhao Zhang, Jia-Wei Liu, Yuchao Gu, Rui Zhao, Weisi Lin, Wynne Hsu, Ying … 展开作者

Jay Zhangjie Wu, Guian Fang, Haoning Wu, Xintao Wang, Yixiao Ge, Xiaodong Cun, David Junhao Zhang, Jia-Wei Liu, Yuchao Gu, Rui Zhao, Weisi Lin, Wynne Hsu, Ying Shan, Mike Zheng Shou

AI总结:

本文针对现有文本到视频生成评估指标在时间维度分析上的不足,提出融合文本-视频对齐度与视频质量两项准则的T2VScore指标,并构建含2543个人工评判的TVGE数据集验证其优越性。

摘要:

生成模型在合成高质量文本、图像和视频方面已展现出卓越能力。在视频生成领域,当代文本到视频模型展现出令人印象深刻的能力,能够创作出视觉上惊艳的视频。然而,评估此类视频仍面临重大挑战。当前研究主要采用FVD、IS和CLIP Score等自动化指标。但这些指标提供的分析并不完整,尤其是在视频内容的时间维度评估方面,因而使其成为真实视频质量不可靠的指示器。此外,尽管用户研究有可能准确反映人类感知,但其耗时费力的特性以及结果常受主观偏差污染的问题制约了其应用。在本文中,我们考察了现有指标固有的局限性,并提出了一种新颖的评估流程——文本到视频评分(T2VScore)。该指标整合了两项关键准则:(1)文本-视频对齐度,用于审视视频在表示给定文本描述时的保真度;(2)视频质量,通过专家混合模型评估视频的整体制作水准。此外,为了评估所提出的指标并促进其未来改进,我们发布了TVGE数据集,收集了针对2543个文本到视频生成视频在上述两项准则上的人工评判。在TVGE数据集上的实验证明了所提出的T2VScore在提供更优文本到视频生成评估指标方面的优越性。

英文摘要:

Generative models have demonstrated remarkable capability in synthesizing high-quality text, images, and videos. For video generation, contemporary text-to-video models exhibit impressive capabilities, crafting visually stunning videos. Nonetheless, evaluating such videos poses significant challenges. Current research predominantly employs automated metrics such as FVD, IS, and CLIP Score. However, these metrics provide an incomplete analysis, particularly in the temporal assessment of video content, thus rendering them unreliable indicators of true video quality. Furthermore, while user studies have the potential to reflect human perception accurately, they are hampered by their time-intensive and laborious nature, with outcomes that are often tainted by subjective bias. In this paper, we investigate the limitations inherent in existing metrics and introduce a novel evaluation pipeline, the Text-to-Video Score (T2VScore). This metric integrates two pivotal criteria: (1) Text-Video Alignment, which scrutinizes the fidelity of the video in representing the given text description, and (2) Video Quality, which evaluates the video's overall production caliber with a mixture of experts. Moreover, to evaluate the proposed metrics and facilitate future improvements on them, we present the TVGE dataset, collecting human judgements of 2,543 text-to-video generated videos on the two criteria. Experiments on the TVGE dataset demonstrate the superiority of the proposed T2VScore on offering a better metric for text-to-video generation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑