Human video generation is a dynamic and rapidly evolving task that aims to synthesize 2D human body video sequences with generative models given control conditions such as text, audio, and pose. With the potential for wide-ranging applications in film, gaming, and virtual communication, the ability to generate natural and realistic human video is critical. Recent advancements in generative models have laid a solid foundation for the growing interest in this area. Despite the significant progress, the task of human video generation remains challenging due to the consistency of characters, the complexity of human motion, and difficulties in their relationship with the environment. This survey provides a comprehensive review of the current state of human video generation, marking, to the best of our knowledge, the first extensive literature review in this domain. We start with an introduction to the fundamentals of human video generation and the evolution of generative models that have facilitated the field's growth. We then examine the main methods employed for three key sub-tasks within human video generation: text-driven, audio-driven, and pose-driven motion generation. These areas are explored concerning the conditions that guide the generation process. Furthermore, we offer a collection of the most commonly utilized datasets and the evaluation metrics that are crucial in assessing the quality and realism of generated videos. The survey concludes with a discussion of the current challenges in the field and suggests possible directions for future research. The goal of this survey is to offer the research community a clear and holistic view of the advancements in human video generation, highlighting the milestones achieved and the challenges that lie ahead.
VABench: A Comprehensive Benchmark for Audio-Video Generation
Peking University(北京大学);Zhongguancun Academy(中关村学院);Beijing Key Laboratory of Data Intelligence (Peking University)(北京市数据智能重点实验室(北京大学));Ant Group(蚂蚁集团);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Huazhong University of Science and Technology(华中科技大学)
"PhyWorldBench": A Comprehensive Evaluation of Physical Realism in Text-to-Video Models
University of California, Santa Cruz(加州大学圣克ruz分校);NVIDIA Research(NVIDIA研究);Northeastern University(东北大学);University of California, Santa Barbara(加州大学圣巴巴拉分校)
VBench: Comprehensive Benchmark Suite for Video Generative Models
Nanyang Technological University(南洋理工大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Nanjing University(南京大学);The Chinese University of Hong Kong(香港中文大学)
VBench++: Comprehensive and Versatile Benchmark Suite for Video Generative Models
Nanyang Technological University(南洋理工大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));Nanjing University(南京大学);The Chinese University of Hong Kong(香港中文大学)
LOVE: Benchmarking and Evaluating Text-to-Video Generation and Video-to-Text Interpretation
Institute of Image Communication and Network Engineering(图像通信与网络工程研究所);MoE Key Lab of Artificial Intelligence(人工智能MoE重点实验室);AI Institute(人工智能研究院);Shanghai Jiao Tong University(上海交通大学)
RobotEQ-Video: A Video-Centric Benchmark for Social Proactive Intelligence with World-State Taxonomy
State Key Laboratory of Autonomous Intelligent Unmanned Systems, Tongji University(同济大学自主智能无人系统全国重点实验室);Tsinghua University(清华大学);The Chinese University of Hong Kong(香港中文大学);Shenzhen MSU-BIT University(深圳北理莫斯科大学);University of Science and Technology of China(中国科学技术大学)
Tencent AI Lab(腾讯AI实验室);Beijing Institute of Technology(北京理工大学);University of California, Berkeley(加州大学伯克利分校);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所)
When SAM2 Meets Video Camouflaged Object Segmentation: A Comprehensive Evaluation and Adaptation
Computer Vision Laboratory, ETH Zürich(苏黎世联邦理工学院计算机视觉实验室);Integrated System Laboratory, ETH Zürich(苏黎世联邦理工学院集成系统实验室);School of Computer Science and Engineering, Nanjing University of Science and Technology(南京理工大学计算机科学与工程学院)
DeepSport: A Multimodal Large Language Model for Comprehensive Sports Video Reasoning via Agentic Reinforcement Learning
Georgia Institute of Technology(佐治亚理工学院);Rice University(Rice大学);Johns Hopkins University(约翰霍普金斯大学);University of California, Irvine(加州大学 Irvine分校);University of California, Santa Barbara(加州大学圣巴巴拉分校)
Shot2Story: A New Benchmark for Comprehensive Understanding of Multi-shot Videos
Bytedance Inc.(字节跳动公司);AAII, UTS(悉尼科技大学人工智能研究院);MBZUAI(穆罕默德·本·扎耶德人工智能大学);University of Science and Technology of China(中国科学技术大学);Data61, CSIRO(澳大利亚联邦科学与工业研究组织数据61部门)
TRECVID 2020: A comprehensive campaign for evaluating video retrieval tasks across multiple application domains
Georgetown University(乔治城大学);National Institute of Standards and Technology(美国国家标准与技术研究院);Johns Hopkins University(约翰斯·霍普金斯大学);MIT Lincoln Laboratory(麻省理工学院林肯实验室);Dublin City University(都柏林城市大学);Trinity College Dublin(都柏林圣三一大学)
OmniMMI: A Comprehensive Multi-modal Interaction Benchmark in Streaming Video Contexts
Beijing Institute for General Artificial Intelligence(北京通用人工智能研究院);State Key Laboratory of General Artificial Intelligence(通用人工智能国家重点实验室);Wangxuan Institute of Computer Technology, Peking University(北京大学王选计算机研究所);X-LANCE Lab, Shanghai Jiao Tong University(上海交通大学X-LANCE实验室)
Video-CoT: A Comprehensive Dataset for Spatiotemporal Understanding of Videos Based on Chain-of-Thought
Institute of Automation, CAS(中国科学院自动化研究所);School of Artifcial Intelligence, UCAS(中国科学技术大学人工智能学院);Beijing Academy of Artificial Intelligence (BAAI)(北京人工智能研究院);Shenzhen International GraduateSchool,Tsinghua University(深圳国际研究生院,清华大学);State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University(多媒体信息处理国家重点实验室,北京大学计算机学院)