Divot：扩散模型驱动的视频Tokenizer用于理解与生成
Divot: Diffusion Powers Video Tokenizer for Comprehension and Generation
AI总结：
针对将图像理解与生成的LLM统一扩展到视频的核心挑战，提出扩散模型驱动的视频Tokenizer Divot，结合预训练LLM形成Divot-Vicuna，在视频理解、生成及storytelling基准上取得有竞争力性能。
摘要：
近年来，将图像理解与生成统一到大型语言模型（LLMs）中的研究兴趣显著激增。这种日益增长的兴趣促使我们探索将这种统一扩展到视频领域。核心挑战在于开发一种多功能的视频Tokenizer，它既能捕捉视频的空间特征，又能捕捉时间动态，以获取供LLMs使用的表示，且这些表示可进一步解码为真实的视频片段以实现视频生成。在这项工作中，我们引入Divot——一种扩散模型驱动的视频Tokenizer，它利用扩散过程进行自监督视频表示学习。我们假设，如果视频扩散模型能以视频Tokenizer的特征为条件有效对视频片段去噪，那么该Tokenizer就成功捕捉了鲁棒的时空信息。此外，视频扩散模型本身可作为解Tokenizer，从表示中解码视频。基于Divot Tokenizer，我们通过视频到文本自回归提出Divot-Vicuna，并通过高斯混合模型对连续值Divot特征的分布建模实现文本到视频生成。实验结果表明，我们的扩散型视频Tokenizer与预训练LLM集成后，在各类视频理解与生成基准上取得了有竞争力的性能。经指令微调的Divot-Vicuna在视频 storytelling 任务中也表现出色，能生成交错的叙事内容及对应的视频。
英文摘要：
In recent years, there has been a significant surge of interest in unifying image comprehension and generation within Large Language Models (LLMs). This growing interest has prompted us to explore extending this unification to videos. The core challenge lies in developing a versatile video tokenizer that captures both the spatial characteristics and temporal dynamics of videos to obtain representations for LLMs, and the representations can be further decoded into realistic video clips to enable video generation. In this work, we introduce Divot, a Diffusion-Powered Video Tokenizer, which leverages the diffusion process for self-supervised video representation learning. We posit that if a video diffusion model can effectively de-noise video clips by taking the features of a video tokenizer as the condition, then the tokenizer has successfully captured robust spatial and temporal information. Additionally, the video diffusion model inherently functions as a de-tokenizer, decoding videos from their representations. Building upon the Divot tokenizer, we present Divot-Vicuna through video-to-text autoregression and text-to-video generation by modeling the distributions of continuous-valued Divot features with a Gaussian Mixture Model. Experimental results demonstrate that our diffusion-based video tokenizer, when integrated with a pre-trained LLM, achieves competitive performance across various video comprehension and generation benchmarks. The instruction tuned Divot-Vicuna also excels in video storytelling, generating interleaved narratives and corresponding videos.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
