预训练扩散模型和 3D 生成领域的最新进展激发了人们对 4D 内容创作的兴趣。然而,实现具有时空一致性的高保真 4D 生成仍然是一个挑战。在这项工作中,我们提出了 STAG4D,这是一种将预训练扩散模型与动态 3D 高斯泼溅相结合以实现高保真 4D 生成的新颖框架。受 3D 生成技术的启发,我们利用多视角扩散模型来初始化锚定于输入视频帧的多视角图像,其中视频可以是真实世界捕获的,也可以是由视频扩散模型生成的。为确保多视角序列初始化的时间一致性,我们引入了一种简单而有效的融合策略,在自注意力计算中利用第一帧作为时间锚点。借助几乎一致的多视角序列,我们随后应用分数蒸馏采样来优化 4D 高斯点云。4D 高斯泼溅专门为生成任务而精心设计,其中提出了一种自适应致密化策略,以缓解不稳定的梯度,从而实现稳健优化。值得注意的是,所提出的流程不需要对扩散网络进行任何预训练或微调,为 4D 生成任务提供了一种更易于使用且实用的解决方案。大量实验表明,我们的方法在渲染质量、时空一致性和生成鲁棒性方面优于先前的 4D 生成工作,为从文本、图像和视频等多样输入进行 4D 生成树立了新的最先进水平。
英文摘要:
Recent progress in pre-trained diffusion models and 3D generation have spurred interest in 4D content creation. However, achieving high-fidelity 4D generation with spatial-temporal consistency remains a challenge. In this work, we propose STAG4D, a novel framework that combines pre-trained diffusion models with dynamic 3D Gaussian splatting for high-fidelity 4D generation. Drawing inspiration from 3D generation techniques, we utilize a multi-view diffusion model to initialize multi-view images anchoring on the input video frames, where the video can be either real-world captured or generated by a video diffusion model. To ensure the temporal consistency of the multi-view sequence initialization, we introduce a simple yet effective fusion strategy to leverage the first frame as a temporal anchor in the self-attention computation. With the almost consistent multi-view sequences, we then apply the score distillation sampling to optimize the 4D Gaussian point cloud. The 4D Gaussian spatting is specially crafted for the generation task, where an adaptive densification strategy is proposed to mitigate the unstable Gaussian gradient for robust optimization. Notably, the proposed pipeline does not require any pre-training or fine-tuning of diffusion networks, offering a more accessible and practical solution for the 4D generation task. Extensive experiments demonstrate that our method outperforms prior 4D generation works in rendering quality, spatial-temporal consistency, and generation robustness, setting a new state-of-the-art for 4D generation from diverse inputs, including text, image, and video.
MSVS-VAE: Multi-Scale Anchored VecSet for High-Fidelity 3D Reconstruction
The Hong Kong University of Science and Technology(香港科技大学);Peking University(北京大学);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));Lightspeed Studios(光速工作室)
National University of Singapore(新加坡国立大学);Nanjing University of Posts and Telecommunications(南京邮电大学);Qilu University of Technology (Shandong Academy of Sciences)(齐鲁工业大学(山东科学院))
Lean-SAM2: Target-Anchored Memory and Encoder Acceleration for SAM2
School of Biomedical Engineering, Hainan University(海南大学生物医学工程学院);Department of Electronics and Electrical Engineering, Keio University(庆应义塾大学电子电气工程系);College of Computer Science, Shenyang Aerospace University(沈阳航空航天大学计算机科学学院);School of Computer Science and Technology, Hainan University(海南大学计算机科学与技术学院)
Event-Anchored Frame Selection for Effective Long-Video Understanding
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University(教育部多媒体可信感知与高效计算重点实验室,厦门大学);College of Computer and Data Science, Fuzhou University(福州大学计算机与数据科学学院)
CapField-OPD: Learning Continuous Capability Fields via Joint-Anchored Multi-Teacher On-Policy Distillation for Flow Models
University of Science and Technology of China(中国科学技术大学);Shanghai Jiao Tong University(上海交通大学);Fudan University(复旦大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室)
Generative Neural Video Compression via Video Diffusion Prior
School of Information and Communication Engineering, Communication University of China(信息与通信工程学院,通信大学);School of Computer Science, Peking University(计算机学院,北京大学)
Physics Encoded Spatial and Temporal Generative Adversarial Network for Tropical Cyclone Image Super-resolution
Nanjing University of Information Science and Technology(南京信息工程大学);University of Reading(reading大学);Shanghai Typhoon Institute, China Meteorological Administration(中国气象局上海台风研究所)
TC-Light: Temporally Coherent Generative Rendering for Realistic World Transfer
NLPR, MAIS, Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);University of Chinese Academy of Sciences(中国科学院大学);Shandong University(山东大学);University of Science and Technology Beijing(北京科技大学);Tencent(腾讯);Huazhong University of Science and Technology(华中科技大学)
ProxyBuild: Text-Guided Structured 3D Building Generation with Mesh-Anchored Procedural Proxies
Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳));Pengcheng Laboratory(鹏城实验室);Harbin Institute of Technology(哈尔滨工业大学);Harbin Institute of Technology, Suzhou Research Institute(哈尔滨工业大学苏州研究院)