Mimir:改进视频扩散模型以提升精准文本理解
Mimir: Improving Video Diffusion Models for Precise Text Understanding
AI总结:
针对视频扩散模型文本理解有限的问题,提出Mimir框架,通过token融合器协调文本编码器与大语言模型(LLMs)输出,利用LLMs优势提升文本理解,生成高质量视频,尤其在短字幕和动态变化场景表现出色。
摘要:
文本因其叙事性成为视频生成中的关键控制信号。为将文本描述转化为视频片段,当前视频扩散模型借鉴文本编码器特征,但文本理解能力有限。大语言模型(LLMs)的近期成功展现了解码器仅架构Transformer的优势,为文本到视频(T2V)生成带来三大益处:卓越可扩展性带来的精准文本理解、下一个token预测实现的超越输入文本的想象力,以及通过指令调优灵活优先考虑用户兴趣。然而,两种不同文本建模范式产生的特征分布差距阻碍了LLMs在现有T2V模型中的直接应用。本研究提出Mimir,一种端到端训练框架,通过精心设计的token融合器协调文本编码器和LLMs的输出。该设计使T2V模型能充分利用已学习的视频先验,同时发挥LLMs的文本相关能力。大量定量和定性结果表明,Mimir在生成高质量视频方面有效,尤其在处理短字幕和管理动态变化时表现出出色的文本理解能力。项目页面:https://lucaria-academy.github.io/Mimir/
英文摘要:
Text serves as the key control signal in video generation due to its narrative nature. To render text descriptions into video clips, current video diffusion models borrow features from text encoders yet struggle with limited text comprehension. The recent success of large language models (LLMs) showcases the power of decoder-only transformers, which offers three clear benefits for text-to-video (T2V) generation, namely, precise text understanding resulting from the superior scalability, imagination beyond the input text enabled by next token prediction, and flexibility to prioritize user interests through instruction tuning. Nevertheless, the feature distribution gap emerging from the two different text modeling paradigms hinders the direct use of LLMs in established T2V models. This work addresses this challenge with Mimir, an end-to-end training framework featuring a carefully tailored token fuser to harmonize the outputs from text encoders and LLMs. Such a design allows the T2V model to fully leverage learned video priors while capitalizing on the text-related capability of LLMs. Extensive quantitative and qualitative results demonstrate the effectiveness of Mimir in generating high-quality videos with excellent text comprehension, especially when processing short captions and managing shifting motions. Project page: https://lucaria-academy.github.io/Mimir/
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。