图像、3D动画和视频的条件生成建模
Conditional Generative Modeling for Images, 3D Animations, and Video
AI总结：
本文提出多种条件生成模型新架构与方法，包括基于Neural ODEs的视频预测、连续归一化流的图像超分辨率、3D角色姿态对齐编辑及非各向高斯去噪扩散视频框架，在多数据集上取得SOTA结果。
摘要：
本论文旨在通过探索条件生成模型的新颖表述，以及在图像、3D动画和视频中的创新应用，推动计算机视觉生成建模领域的创新。我们的研究侧重于提供噪声与视觉数据可逆变换的架构，以及将编码器-解码器架构应用于生成任务和3D内容操作。在所有实例中，我们均结合条件信息来增强视觉数据的合成，从而提升生成过程的效率及生成内容的质量。我们引入Neural ODEs，利用编码器-解码器架构对视频动态进行建模，展示了其仅在训练重建当前帧的情况下仍能预测未来视频帧的能力。接着，我们提出连续归一化流的条件变体，能够基于低分辨率输入生成更高分辨率的图像，在减少参数和训练时间的同时实现了可比的图像质量。我们的下一项贡献提出了一个流水线，以人类图像作为输入，自动将用户指定的3D角色与人类姿态对齐，并支持基于部分输入的姿态编辑。随后，我们推导了使用非各向同性高斯过程的去噪扩散模型的相关数学细节，并展示了可比的生成质量。最后，我们设计了一个新颖的去噪扩散框架，能够解决预测、生成和插值这三个视频任务。我们进行了消融研究，并在多个数据集上展示了SOTA结果。我们的贡献已发表于同行评审会议的论文中。总体而言，我们的研究旨在为追求更高效和灵活的生成模型做出有意义的贡献，并有望塑造计算机视觉的未来。
英文摘要：
This dissertation attempts to drive innovation in the field of generative modeling for computer vision, by exploring novel formulations of conditional generative models, and innovative applications in images, 3D animations, and video. Our research focuses on architectures that offer reversible transformations of noise and visual data, and the application of encoder-decoder architectures for generative tasks and 3D content manipulation. In all instances, we incorporate conditional information to enhance the synthesis of visual data, improving the efficiency of the generation process as well as the generated content. We introduce the use of Neural ODEs to model video dynamics using an encoder-decoder architecture, demonstrating their ability to predict future video frames despite being trained solely to reconstruct current frames. Next, we propose a conditional variant of continuous normalizing flows that enables higher-resolution image generation based on lower-resolution input, achieving comparable image quality while reducing parameters and training time. Our next contribution presents a pipeline that takes human images as input, automatically aligns a user-specified 3D character with the pose of the human, and facilitates pose editing based on partial inputs. Next, we derive the relevant mathematical details for denoising diffusion models that use non-isotropic Gaussian processes, and show comparable generation quality. Finally, we devise a novel denoising diffusion framework capable of solving all three video tasks of prediction, generation, and interpolation. We perform ablation studies, and show SOTA results on multiple datasets. Our contributions are published articles at peer-reviewed venues. Overall, our research aims to make a meaningful contribution to the pursuit of more efficient and flexible generative models, with the potential to shape the future of computer vision.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
