DreamStory：通过LLM引导的多主体一致性扩散模型实现开放域故事可视化
DreamStory: Open-Domain Story Visualization by LLM-Guided Multi-Subject Consistent Diffusion
AI总结：
提出DreamStory框架，通过LLM生成提示词与多模态锚点，结合含掩码互注意力模块的多主体一致性扩散模型MSD，解决开放域故事可视化中多主体连贯生成的难题，并构建DS-500基准验证其有效性。
摘要：
故事可视化旨在创作与文本叙事相对应的引人入胜的图像或视频。尽管扩散模型的最新进展取得了可喜的成果，但现有方法仍然难以仅凭故事创作出连贯的主体一致的画面序列。为此，我们提出DreamStory，一个利用LLM和新型多主体一致性扩散模型的自动开放域故事可视化框架。DreamStory包含（1）充当故事导演的LLM，以及（2）用于在图像间生成一致多主体的创新多主体一致性扩散模型（MSD）。首先，DreamStory利用LLM生成与故事对齐的主体和场景描述性提示词，并为每个场景的主体进行标注，以便后续的主体一致性生成。其次，DreamStory利用这些详细的主体描述创建主体肖像，这些肖像及其对应的文本信息作为多模态锚点（指导）。最后，MSD利用这些多模态锚点生成具有一致多主体的故事场景。具体而言，MSD包含掩码互自注意力（MMSA）和掩码互交叉注意力（MMCA）模块。MMSA和MMCA模块分别确保与参考图像和文本的外观与语义一致性。这两个模块均采用掩码机制来防止主体融合。为了验证我们的方法并推动故事可视化的进展，我们建立了一个基准DS-500，它可以评估故事可视化框架的整体性能、主体识别准确率以及生成模型的一致性。大量实验验证了DreamStory在主观和客观评估中的有效性。请访问我们的项目主页https://dream-xyz.github.io/dreamstory。
英文摘要：
Story visualization aims to create visually compelling images or videos corresponding to textual narratives. Despite recent advances in diffusion models yielding promising results, existing methods still struggle to create a coherent sequence of subject-consistent frames based solely on a story. To this end, we propose DreamStory, an automatic open-domain story visualization framework by leveraging the LLMs and a novel multi-subject consistent diffusion model. DreamStory consists of (1) an LLM acting as a story director and (2) an innovative Multi-Subject consistent Diffusion model (MSD) for generating consistent multi-subject across the images. First, DreamStory employs the LLM to generate descriptive prompts for subjects and scenes aligned with the story, annotating each scene's subjects for subsequent subject-consistent generation. Second, DreamStory utilizes these detailed subject descriptions to create portraits of the subjects, with these portraits and their corresponding textual information serving as multimodal anchors (guidance). Finally, the MSD uses these multimodal anchors to generate story scenes with consistent multi-subject. Specifically, the MSD includes Masked Mutual Self-Attention (MMSA) and Masked Mutual Cross-Attention (MMCA) modules. MMSA and MMCA modules ensure appearance and semantic consistency with reference images and text, respectively. Both modules employ masking mechanisms to prevent subject blending. To validate our approach and promote progress in story visualization, we established a benchmark, DS-500, which can assess the overall performance of the story visualization framework, subject-identification accuracy, and the consistency of the generation model. Extensive experiments validate the effectiveness of DreamStory in both subjective and objective evaluations. Please visit our project homepage at https://dream-xyz.github.io/dreamstory.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
