arXivDaily arXiv每日学术速递 周一至周五更新
稍安勿躁,官方尚未更新
arXiv 2606.02441cs.CV
收藏

空间-时间解耦参考条件用于身份保持的文本到视频生成

Spatial-Temporal Decoupled Reference Conditioning for Identity-Preserving Text-to-Video Generation

Shanghai Jiao Tong University(上海交通大学);University of Electronic Science and Technology of China(电子科技大学);Zhejiang University(浙江大学)

Yuheng Chen, Teng Hu, Yuji Wang, Qingdong He, Lizhuang Ma, Jiangning Zhang

AI总结:

提出ST-DRC框架,通过空间-时间解耦参考条件、TASS-RoPE机制和身份目标,实现高保真身份保持视频生成。

摘要:

身份保持视频生成(IPVG)旨在合成高保真视频,遵循文本提示同时忠实保持参考身份。尽管最近取得进展,现有IPVG方法仍难以平衡高级语义控制和低级身份保真度。为弥合这一差距,我们提出ST-DRC,一种有效的空间-时间解耦参考条件框架,用于身份保持的文本到视频生成。在框架层面,ST-DRC通过使用视频VAE编码参考图像并将其与噪声视频潜在变量拼接,执行潜在上下文特征注入,无需额外适配器即可访问丰富的低级身份细节。为将身份感知参考检索与外观复制分离,我们引入TASS-RoPE,一种时间相邻-空间偏移的RoPE方案,将参考令牌在时间上靠近视频序列但在空间上偏移,允许参考信息通过时空注意力流动,同时抑制像素级复制粘贴捷径。为进一步防止捷径学习并增强扩散目标中被稀释的身份监督,我们结合外观不变参考增强与面部引导身份目标,鼓励模型在颜色、姿态和布局变化下保持身份。在推理时,我们引入三流参考无分类器引导策略,独立控制文本遵循度和参考保真度。实验表明,ST-DRC在基于LTX-2.3的轻量级设计下,实现了强身份保持、提示对齐、时间一致性和视频质量。我们的方法在面部身份保持视频生成赛道中排名靠前,验证了空间-时间解耦参考条件的有效性。

英文摘要:

Identity-preserving video generation (IPVG) aims to synthesize high-fidelity videos that follow text prompts while faithfully preserving a reference identity. Despite recent progress, existing IPVG methods still struggle to balance high-level semantic control and low-level identity fidelity. To bridge this gap, we propose ST-DRC, an effective Spatial-Temporal Decoupled Reference Conditioning framework for identity-preserving text-to-video generation. At the framework level, ST-DRC performs latent in-context feature injection by encoding the reference image with the video VAE and concatenating it with noisy video latents, enabling rich low-level identity details to be accessed without additional adapters. To separate identity-aware reference retrieval from appearance copying, we introduce TASS-RoPE, a Temporal-Adjacent Spatial-Shifted RoPE scheme that places reference tokens near the video sequence in time but shifts them in space, allowing reference information to flow through spatio-temporal attention while suppressing pixel-level copy-paste shortcuts. To further prevent shortcut learning and strengthen the otherwise diluted identity supervision in the diffusion objective, we combine appearance-invariant reference augmentation with face-guided identity objectives, encouraging the model to preserve identity under variations in color, pose, and layout. At inference time, we introduce a three-stream reference classifier-free guidance strategy that independently controls text adherence and reference fidelity. Experiments demonstrate that ST-DRC achieves strong identity preservation, prompt alignment, temporal consistency, and video quality with a lightweight design built on LTX-2.3. Our method ranks among the top submissions in the facial identity-preserving video generation track, validating the effectiveness of spatial-temporal decoupled reference conditioning.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑