arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2409.19624cs.CVcs.AI
收藏

Storynizor:通过帧间同步与随机打乱 ID 注入实现一致性故事生成

Storynizor: Consistent Story Generation via Inter-Frame Synchronized and Shuffled ID Injection

Fuxi AI Lab, NetEase Inc.(网易伏羲人工智能实验室);Multimedia Laboratory, The Chinese University of Hong Kong(香港中文大学多媒体实验室)

Yuhang Ma, Wenting Xu, Chaoyi Zhao, Keqiang Sun, Qinfeng Jin, Zeng Zhao, Changjie Fan, Zhipeng Hu

AI总结:

本文提出 Storynizor 模型,通过 ID-Synchronizer 和 ID-Injector 两个关键模块及 StoryDB 数据集,实现高保真角色一致性、灵活姿态和生动背景的连贯故事图像生成。

摘要:

近年来,文本到图像扩散模型的进展引发了人们对连续故事图像生成的浓厚兴趣。本文提出了 Storynizor,该模型能够生成连贯的故事,具备较强的帧间角色一致性、有效的前景与背景分离以及多样的姿态变化。Storynizor 的核心创新在于其关键模块:ID-Synchronizer 和 ID-Injector。ID-Synchronizer 采用自动掩码自注意力模块和跨帧图像间的掩码感知损失,以提升角色生成的一致性,生动地表现角色的姿态和背景。ID-Injector 利用 Shuffling Reference Strategy(SRS)将 ID 特征整合到特定位置,从而增强基于 ID 的一致角色生成。此外,为便于训练 Storynizor,我们构建了一个名为 StoryDB 的新数据集,包含 100,000 张图像。该数据集包含处于多样环境、布局和姿态中的单角色和多角色集合,并带有详细描述。实验结果表明,与其他针对特定角色的方法相比,Storynizor 展现出卓越的连贯故事生成能力,具有高保真的角色一致性、灵活的姿态和生动的背景。

英文摘要:

Recent advances in text-to-image diffusion models have spurred significant interest in continuous story image generation. In this paper, we introduce Storynizor, a model capable of generating coherent stories with strong inter-frame character consistency, effective foreground-background separation, and diverse pose variation. The core innovation of Storynizor lies in its key modules: ID-Synchronizer and ID-Injector. The ID-Synchronizer employs an auto-mask self-attention module and a mask perceptual loss across inter-frame images to improve the consistency of character generation, vividly representing their postures and backgrounds. The ID-Injector utilize a Shuffling Reference Strategy (SRS) to integrate ID features into specific locations, enhancing ID-based consistent character generation. Additionally, to facilitate the training of Storynizor, we have curated a novel dataset called StoryDB comprising 100, 000 images. This dataset contains single and multiple-character sets in diverse environments, layouts, and gestures with detailed descriptions. Experimental results indicate that Storynizor demonstrates superior coherent story generation with high-fidelity character consistency, flexible postures, and vivid backgrounds compared to other character-specific methods.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑