arXiv 2502.065272025-02-20cs.CV

CustomVideoX：面向零样本定制视频扩散 Transformer 的三维参考注意力驱动动态适配

CustomVideoX: 3D Reference Attention Driven Dynamic Adaptation for Zero-Shot Customized Video Diffusion Transformers

D. She, Mushui Liu, Jingxuan Pang, Jin Wang, Zhen Yang, Wanggui He, Guanghao Zhang, Yi Wang, Qihan Huang, Haobin Tang, Yunlong Yu, Siming Fu

备注：: Section 4 in CustomVideoX Entity Region-Aware Enhancement has description errors. The compared methods data of Table I lacks other metrics

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出 CustomVideoX 框架，通过仅训练 LoRA、3D Reference Attention、TAB 与 ERAE 实现参考图像驱动的零样本个性化视频生成，并构建 VideoBench 验证其一致性和质量优势。

摘要：

定制化生成已在图像合成中取得显著进展，但由于时间不一致性和质量退化，个性化视频生成仍具挑战。本文提出 CustomVideoX，这是一个利用视频扩散 Transformer 从参考图像生成个性化视频的创新框架。CustomVideoX 借助预训练视频网络，仅训练 LoRA 参数以提取参考特征，从而兼顾效率与适应性。为促进参考图像与视频内容之间的无缝交互，我们提出 3D Reference Attention，使参考图像特征能够沿空间和时间维度直接、同时地与所有视频帧交互。为缓解推理阶段参考图像特征和文本引导对生成视频内容的过度影响，我们实施 Time-Aware Reference Attention Bias（TAB）策略，在不同时间步动态调节参考偏置。此外，我们引入 Entity Region-Aware Enhancement（ERAE）模块，通过调整注意力偏置，将关键实体 token 的高激活区域与参考特征注入对齐。为全面评估个性化视频生成，我们建立了一个新基准 VideoBench，包含 50 多个对象和 100 条提示，用于广泛评测。实验结果表明，CustomVideoX 在视频一致性和质量方面显著优于现有方法。

英文摘要：

Customized generation has achieved significant progress in image synthesis, yet personalized video generation remains challenging due to temporal inconsistencies and quality degradation. In this paper, we introduce CustomVideoX, an innovative framework leveraging the video diffusion transformer for personalized video generation from a reference image. CustomVideoX capitalizes on pre-trained video networks by exclusively training the LoRA parameters to extract reference features, ensuring both efficiency and adaptability. To facilitate seamless interaction between the reference image and video content, we propose 3D Reference Attention, which enables direct and simultaneous engagement of reference image features with all video frames across spatial and temporal dimensions. To mitigate the excessive influence of reference image features and textual guidance on generated video content during inference, we implement the Time-Aware Reference Attention Bias (TAB) strategy, dynamically modulating reference bias over different time steps. Additionally, we introduce the Entity Region-Aware Enhancement (ERAE) module, aligning highly activated regions of key entity tokens with reference feature injection by adjusting attention bias. To thoroughly evaluate personalized video generation, we establish a new benchmark, VideoBench, comprising over 50 objects and 100 prompts for extensive assessment. Experimental results show that CustomVideoX significantly outperforms existing methods in terms of video consistency and quality.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。