Aura：通过基于VLM的语义对齐实现一致的多主体视频生成
Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment
AI总结：
提出Aura框架用于高保真和身份一致的视频生成，引入AI导演级字幕、利用VLM提取多模态语义特征，设计对齐策略，采用多种机制减少伪影，在多主体视频生成上达先进性能。
摘要：
主题驱动和多元素视频生成是可控视频合成的核心，但现有方法在保持身份一致性和建模多主体复杂关系方面仍有困难。本文提出Aura，一个用于高保真和身份一致视频生成的统一框架。为更好捕捉场景动态和主体交互，引入AI导演级字幕提供密集结构化视频内容描述。利用带可学习查询的视觉语言模型从文本和视觉参考中提取多模态语义特征。设计两阶段对齐策略将VLM特征映射到DiT特征空间。采用令牌连接进行视觉条件处理，开发主体感知RoPE-Shift机制等减少伪影。推理时，Progressive-APG减轻过饱和并改善与用户提示的语义对齐。最后构建高质量视频-主体图像数据集。大量实验表明该方法在单主体生成和更具挑战性的多元素场景中均实现了领先性能。
英文摘要：
Subject-driven and multi-element video generation are central to controllable video synthesis, but existing methods still struggle to preserve identity consistency and model complex relationships among multiple subjects. In this paper, we propose Aura, a unified framework for high-fidelity and identity-consistent video generation. To better capture scene dynamics and subject interactions, we introduce AI director-level captions that provide dense and structured descriptions of video content. We further leverage a vision-language model (VLM) with learnable queries to extract multimodal semantic features from textual and visual references, covering both global semantics and fine-grained visual cues. To bridge the representational gap between the VLM and the Diffusion Transformer (DiT), we design a two-stage alignment strategy that progressively maps VLM features into the DiT feature space. For visual conditioning, we adopt token concatenation to inject reference information directly into the generation process. To distinguish heterogeneous subject types and reduce common copy-paste artifacts, we develop a subject-aware RoPE-Shift mechanism. To further differentiate reference images of different categories, we introduce subject-aware learnable tokens. In addition, we introduce Memory Tokens to balance the training signal across examples with different numbers of reference subjects. During inference, Progressive-APG (Adaptive Prompt Guidance) further alleviates oversaturation and improves semantic alignment with user prompts. Finally, we build a high-quality video-subject image dataset through a dedicated data construction pipeline. Extensive experiments show that our method achieves state-of-the-art performance on both single-subject generation and more challenging multi-element scenarios.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
