arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2403.09669cs.CVcs.AI
收藏

STREAM:面向视频生成模型的时空评估与分析指标

STREAM: Spatio-TempoRal Evaluation and Analysis Metric for Video Generative Models

Ulsan National Institute of Science and Technology(蔚山科学技术院)

Pum Jun Kim, Seojun Kim, Jaejun Yoo

AI总结:

针对现有视频评估指标偏重空间、受帧数限制且不稳定等问题,提出STREAM指标,首次实现独立评估视频空间与时间质量,不受视频长度约束,为视频生成模型提供更全面的改进洞见。

摘要:

图像生成模型在生成逼真且多样的图像方面取得了显著进展,并得到了各种评估指标的全面指导支持。然而,当前的视频生成模型甚至难以生成长度很短的视频片段,且能够为改进提供洞见的工具十分有限。当前的视频评估指标只是图像指标的简单改编,仅将嵌入网络替换为视频嵌入网络,这可能低估了视频的独特特性。我们的分析表明,广泛使用的Fréchet Video Distance(FVD)对空间方面的重视程度高于视频的时间自然性,并且本质上受限于所用嵌入网络的输入尺寸,将其限制在16帧以内。此外,FVD表现出相当大的不稳定性,并与人类评估结果存在偏差。为了解决这些局限性,我们提出了STREAM,这是一种新的视频评估指标,专门设计用于独立评估空间和时间方面。这一特性允许从多种视角对视频生成模型进行全面分析和评估,且不受视频长度限制。我们提供了分析和实验证据,证明STREAM为视频的视觉质量和时间质量提供了有效的评估工具,并为视频生成模型的改进方向提供了洞见。据我们所知,STREAM是第一个能够分别评估视频时间方面和空间方面的评估指标。我们的代码可在 https://github.com/pro2nit/STREAM 获取。

英文摘要:

Image generative models have made significant progress in generating realistic and diverse images, supported by comprehensive guidance from various evaluation metrics. However, current video generative models struggle to generate even short video clips, with limited tools that provide insights for improvements. Current video evaluation metrics are simple adaptations of image metrics by switching the embeddings with video embedding networks, which may underestimate the unique characteristics of video. Our analysis reveals that the widely used Frechet Video Distance (FVD) has a stronger emphasis on the spatial aspect than the temporal naturalness of video and is inherently constrained by the input size of the embedding networks used, limiting it to 16 frames. Additionally, it demonstrates considerable instability and diverges from human evaluations. To address the limitations, we propose STREAM, a new video evaluation metric uniquely designed to independently evaluate spatial and temporal aspects. This feature allows comprehensive analysis and evaluation of video generative models from various perspectives, unconstrained by video length. We provide analytical and experimental evidence demonstrating that STREAM provides an effective evaluation tool for both visual and temporal quality of videos, offering insights into area of improvement for video generative models. To the best of our knowledge, STREAM is the first evaluation metric that can separately assess the temporal and spatial aspects of videos. Our code is available at https://github.com/pro2nit/STREAM.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑