TaleCrafter：多角色交互式故事可视化
TaleCrafter: Interactive Story Visualization with Multiple Characters
AI总结：
本文提出 TaleCrafter 交互式故事可视化系统，通过 S2P、T2L、C-T2I 和 I2V 四个组件支持多角色、布局编辑与动画生成，并验证了其有效性和灵活性。
摘要：
准确的故事可视化需要若干必要要素，例如跨帧的身份一致性、纯文本与视觉内容之间的对齐，以及图像中对象的合理布局。以往大多数工作试图通过在同一风格且具有相同角色的视频集（例如 FlintstonesSV 数据集）上拟合文本到图像（T2I）模型来满足这些要求。然而，学习得到的 T2I 模型通常难以适应新角色、新场景和新风格，并且往往缺乏修改合成图像布局的灵活性。本文提出了一个面向通用交互式故事可视化的系统，能够处理多个新角色，并支持对布局和局部结构进行编辑。该系统通过利用在大规模语料上训练的大型语言模型和 T2I 模型的先验知识而开发，由四个相互关联的组件组成：故事到提示生成（S2P）、文本到布局生成（T2L）、可控文本到图像生成（C-T2I）以及图像到视频动画（I2V）。首先，S2P 模块将简洁的故事信息转换为后续阶段所需的详细提示。接着，T2L 基于提示生成多样且合理的布局，并为用户提供根据偏好调整和细化布局的能力。核心组件 C-T2I 支持在布局、草图和演员特定标识符的引导下创建图像，以保持可视化过程中的一致性和细节。最后，I2V 通过对生成图像进行动画化来丰富可视化过程。本文进行了大量实验和用户研究，以验证所提系统在交互式编辑方面的有效性和灵活性。
英文摘要：
Accurate Story visualization requires several necessary elements, such as identity consistency across frames, the alignment between plain text and visual content, and a reasonable layout of objects in images. Most previous works endeavor to meet these requirements by fitting a text-to-image (T2I) model on a set of videos in the same style and with the same characters, e.g., the FlintstonesSV dataset. However, the learned T2I models typically struggle to adapt to new characters, scenes, and styles, and often lack the flexibility to revise the layout of the synthesized images. This paper proposes a system for generic interactive story visualization, capable of handling multiple novel characters and supporting the editing of layout and local structure. It is developed by leveraging the prior knowledge of large language and T2I models, trained on massive corpora. The system comprises four interconnected components: story-to-prompt generation (S2P), text-to-layout generation (T2L), controllable text-to-image generation (C-T2I), and image-to-video animation (I2V). First, the S2P module converts concise story information into detailed prompts required for subsequent stages. Next, T2L generates diverse and reasonable layouts based on the prompts, offering users the ability to adjust and refine the layout to their preference. The core component, C-T2I, enables the creation of images guided by layouts, sketches, and actor-specific identifiers to maintain consistency and detail across visualizations. Finally, I2V enriches the visualization process by animating the generated images. Extensive experiments and a user study are conducted to validate the effectiveness and flexibility of interactive editing of the proposed system.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
