HunyuanCustom：面向定制化视频生成的多模态驱动架构
HunyuanCustom: A Multimodal-Driven Architecture for Customized Video Generation
AI总结：
HunyuanCustom提出一种多模态定制化视频生成框架，通过文本-图像融合、图像ID增强及音频/视频条件注入机制，在保持主体一致性的同时显著提升生成质量。
摘要：
定制化视频生成旨在根据用户灵活定义的条件生成包含特定主体的视频，但现有方法往往在身份一致性和输入模态的有限性方面存在不足。本文提出了HunyuanCustom，一个多模态定制化视频生成框架，强调主体一致性，同时支持图像、音频、视频和文本条件。该模型构建于HunyuanVideo之上，首先通过引入基于LLaVA的文本-图像融合模块来增强多模态理解，以处理图像-文本条件生成任务；同时引入图像ID增强模块，利用时间拼接在帧间强化身份特征。为实现音频和视频条件生成，我们进一步提出了模态特定的条件注入机制：AudioNet模块通过空间交叉注意力实现层次化对齐；视频驱动注入模块通过基于patchify的特征对齐网络整合潜在压缩的条件视频。在单主体和多主体场景上的大量实验表明，HunyuanCustom在ID一致性、真实感和文本-视频对齐方面显著优于最先进的开源和闭源方法。此外，我们验证了其在下游任务中的鲁棒性，包括音频和视频驱动的定制化视频生成。我们的结果凸显了多模态条件控制和身份保持策略在推进可控视频生成方面的有效性。所有代码和模型均可在https://hunyuancustom.github.io获取。
英文摘要：
Customized video generation aims to produce videos featuring specific subjects under flexible user-defined conditions, yet existing methods often struggle with identity consistency and limited input modalities. In this paper, we propose HunyuanCustom, a multi-modal customized video generation framework that emphasizes subject consistency while supporting image, audio, video, and text conditions. Built upon HunyuanVideo, our model first addresses the image-text conditioned generation task by introducing a text-image fusion module based on LLaVA for enhanced multi-modal understanding, along with an image ID enhancement module that leverages temporal concatenation to reinforce identity features across frames. To enable audio- and video-conditioned generation, we further propose modality-specific condition injection mechanisms: an AudioNet module that achieves hierarchical alignment via spatial cross-attention, and a video-driven injection module that integrates latent-compressed conditional video through a patchify-based feature-alignment network. Extensive experiments on single- and multi-subject scenarios demonstrate that HunyuanCustom significantly outperforms state-of-the-art open- and closed-source methods in terms of ID consistency, realism, and text-video alignment. Moreover, we validate its robustness across downstream tasks, including audio and video-driven customized video generation. Our results highlight the effectiveness of multi-modal conditioning and identity-preserving strategies in advancing controllable video generation. All the code and models are available at https://hunyuancustom.github.io.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
