arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2501.04698cs.CV

ConceptMaster:基于扩散Transformer模型且无需测试时调优的多概念视频定制

ConceptMaster: Multi-Concept Video Customization on Diffusion Transformer Models Without Test-Time Tuning

Sun Yat-sen University(中山大学);Kuaishou Technology(快手科技);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Tsinghua University(清华大学)

Yuzhou Huang, Ziyang Yuan, Quande Liu, Qiulin Wang, Xintao Wang, Ruimao Zhang, Pengfei Wan, Di Zhang, Kun Gai

AI总结:

针对多概念视频定制中的身份混淆与高质量数据稀缺问题,提出无需测试时调优的 ConceptMaster,通过独立注入解耦嵌入并构建数据与评估流程,显著提升多身份视频生成质量。

摘要:

文本到视频生成已通过扩散模型取得显著进展。然而,多概念视频定制(MCVC)仍是一项重大挑战。我们识别出该任务的两个关键挑战:1)身份解耦问题,即直接采用现有定制方法在同时处理多个概念时不可避免地会混淆身份属性;2)高质量视频-实体对稀缺,而这类数据对于训练一个能够在视频生成中良好表示并解耦各种定制概念的模型至关重要。为应对这些挑战,我们提出 ConceptMaster,这是一个新颖框架,可有效解决身份解耦问题,同时在视频定制中保持概念保真度。具体而言,我们提出学习解耦的多概念嵌入,并以独立方式将其注入扩散模型,从而有效保证具有多个身份的定制视频质量,即使对于高度相似的视觉概念也是如此。为克服高质量 MCVC 数据稀缺的问题,我们建立了一条数据构建流程,能够在多样化场景中收集高质量的多概念视频-实体数据对。我们还进一步设计了一个多概念视频评估集,从概念保真度、身份解耦能力和视频生成质量三个维度,在六种不同的概念组合场景中全面验证我们的方法。大量实验表明,ConceptMaster 在视频定制任务上显著优于以往方法,展现出为视频扩散模型生成个性化且语义准确内容的巨大潜力。

英文摘要:

Text-to-video generation has made remarkable advancements through diffusion models. However, Multi-Concept Video Customization (MCVC) remains a significant challenge. We identify two key challenges for this task: 1) the identity decoupling issue, where directly adopting existing customization methods inevitably mix identity attributes when handling multiple concepts simultaneously, and 2) the scarcity of high-quality video-entity pairs, which is crucial for training a model that can well represent and decouple various customized concepts in video generation. To address these challenges, we introduce ConceptMaster, a novel framework that effectively addresses the identity decoupling issues while maintaining concept fidelity in video customization. Specifically, we propose to learn decoupled multi-concept embeddings and inject them into diffusion models in a standalone manner, which effectively guarantees the quality of customized videos with multiple identities, even for highly similar visual concepts. To overcome the scarcity of high-quality MCVC data, we establish a data construction pipeline, which enables collection of high-quality multi-concept video-entity data pairs across diverse scenarios. A multi-concept video evaluation set is further devised to comprehensively validate our method from three dimensions, including concept fidelity, identity decoupling ability, and video generation quality, across six different concept composition scenarios. Extensive experiments demonstrate that ConceptMaster significantly outperforms previous methods for video customization tasks, showing great potential to generate personalized and semantically accurate content for video diffusion models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑