Generative models have demonstrated remarkable capability in synthesizing high-quality text, images, and videos. For video generation, contemporary text-to-video models exhibit impressive capabilities, crafting visually stunning videos. Nonetheless, evaluating such videos poses significant challenges. Current research predominantly employs automated metrics such as FVD, IS, and CLIP Score. However, these metrics provide an incomplete analysis, particularly in the temporal assessment of video content, thus rendering them unreliable indicators of true video quality. Furthermore, while user studies have the potential to reflect human perception accurately, they are hampered by their time-intensive and laborious nature, with outcomes that are often tainted by subjective bias. In this paper, we investigate the limitations inherent in existing metrics and introduce a novel evaluation pipeline, the Text-to-Video Score (T2VScore). This metric integrates two pivotal criteria: (1) Text-Video Alignment, which scrutinizes the fidelity of the video in representing the given text description, and (2) Video Quality, which evaluates the video's overall production caliber with a mixture of experts. Moreover, to evaluate the proposed metrics and facilitate future improvements on them, we present the TVGE dataset, collecting human judgements of 2,543 text-to-video generated videos on the two criteria. Experiments on the TVGE dataset demonstrate the superiority of the proposed T2VScore on offering a better metric for text-to-video generation.
In-Style: Bridging Text and Uncurated Videos with Style Transfer for Text-Video Retrieval
Goethe University Frankfurt(法兰克福大学);Max-Planck-Institute for Informatics(马克斯·普朗克信息学研究所);University of Bonn(波恩大学);MIT-IBM Watson AI Lab(麻省理工学院-IBM沃森人工智能实验室)
VideoDreamer: Customized Multi-Subject Text-to-Video Generation with Disen-Mix Finetuning on Language-Video Foundation Models
Tsinghua University(清华大学);Beijing National Research Center for Information Science and Technology(北京信息科学与技术国家研究中心);Beijing Film Academy(北京电影学院);Beijing University of Technology(北京工业大学)
CoCoCo: Improving Text-Guided Video Inpainting for Better Consistency, Controllability and Compatibility
The Chinese University of Hong Kong(香港中文大学);International Digital Economy Academy (IDEA)(国际数字经济学院);The University of Hong Kong(香港大学);Tsinghua University(清华大学)
Gen-L-Video: Multi-Text to Long Video Generation via Temporal Co-Denoising
Shanghai AI Laboratory(上海人工智能实验室);The Chinese University of Hong Kong(香港中文大学);Nanjing University(南京大学);Tsinghua University(清华大学);SenseTime Research(商汤科技研究院)
Recurrent Deconvolutional Generative Adversarial Networks with Application to Text Guided Video Generation
University of Chinese Academy of Sciences(中国科学院大学);The Institute of Microelectronics, Tsinghua University(清华大学微电子研究所);Chinese Academy of Sciences Artificial Intelligence Research(中国科学院人工智能研究院)
LOVE: Benchmarking and Evaluating Text-to-Video Generation and Video-to-Text Interpretation
Institute of Image Communication and Network Engineering(图像通信与网络工程研究所);MoE Key Lab of Artificial Intelligence(人工智能MoE重点实验室);AI Institute(人工智能研究院);Shanghai Jiao Tong University(上海交通大学)
Future Video Generation Better Aligns with the Human Visual Cortex than Observed Video
Yonsei University College of Medicine(延世大学医学院);Institute of Behavioral Sciences in Medicine, Yonsei University College of Medicine(延世大学医学院行为科学研究所);Department of Biomedical Systems Informatics, Yonsei University College of Medicine(延世大学医学院生物医学系统信息学系);Korea University(高丽大学);Yonsei Institute for Digital Healthcare, Yonsei University(延世大学数字医疗研究所)
VG-TVP: Multimodal Procedural Planning via Visually Grounded Text-Video Prompting
National University of Singapore(新加坡国立大学);Institute for Infocomm Research, Agency for Science, Technology, and Research (A*STAR)(新加坡科技研究局(A*STAR)信息通信研究院)
Consistent Zero-shot 3D Texture Synthesis Using Geometry-aware Diffusion and Temporal Video Models
Department of Image, Chung-Ang University(Chung-Ang大学图像系);Graphics AI Lab, NCSOFT(NCSOFT图形AI实验室);Department of Computer Science, University of Seoul(首尔大学计算机科学系)
RichSpace: Enriching Text-to-Video Prompt Space via Text Embedding Interpolation
Zhejiang University(浙江大学);The University of Texas at Austin(德克萨斯大学奥斯汀分校);University of New South Wales(新南威尔士大学);The University of Hong Kong(香港大学);University of Wisconsin-Madison(威斯康星大学麦迪逊分校);Tsinghua University(清华大学);Simons Institute for the Theory of Computing, University of California, Berkeley(加州大学伯克利分校西蒙斯理论计算研究所)
Sounding Video Generator: A Unified Framework for Text-guided Sounding Video Generation
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院)