StoryGPT-V：大型语言模型作为一致的故事可视化器
StoryGPT-V: Large Language Models as Consistent Story Visualizers
AI总结：
StoryGPT-V结合潜在扩散模型和大型语言模型，通过角色感知LDM和LLM对齐，解决故事可视化中的指代消解和角色一致性问题，在两个基准上取得优越结果。
摘要：
近期生成模型在基于文本提示生成逼真且视觉愉悦的图像方面展现了令人印象深刻的能力。然而，将这些模型应用于更复杂的故事可视化任务仍面临重大挑战，因为该任务需要解决帧描述中的代词（他、她、他们）指代问题，即指代消解，并确保跨帧的一致角色和背景合成。新兴的大型语言模型（LLM）展现出强大的推理能力，能够处理模糊指代并处理长序列。因此，我们提出了StoryGPT-V，它利用潜在扩散模型（LDM）和LLM的优势，基于给定故事描述生成具有一致且高质量角色的图像。首先，我们训练一个角色感知的LDM，该模型以角色增强的语义嵌入作为输入，并使用角色分割掩码对交叉注意力图进行监督，旨在提高角色生成的准确性和忠实度。在第二阶段，我们实现了LLM输出与第一阶段模型输入空间中角色增强嵌入之间的对齐。这利用了LLM的推理能力来解决模糊指代，以及理解能力来记忆上下文。我们在两个视觉故事可视化基准上进行了全面实验。我们的模型报告了优越的定量结果，并始终生成具有显著质量且内存消耗低的准确角色。我们的代码公开于：https://xiaoqian-shen.github.io/StoryGPT-V。
英文摘要：
Recent generative models have demonstrated impressive capabilities in generating realistic and visually pleasing images grounded on textual prompts. Nevertheless, a significant challenge remains in applying these models for the more intricate task of story visualization. Since it requires resolving pronouns (he, she, they) in the frame descriptions, i.e., anaphora resolution, and ensuring consistent characters and background synthesis across frames. Yet, the emerging Large Language Model (LLM) showcases robust reasoning abilities to navigate through ambiguous references and process extensive sequences. Therefore, we introduce \emph{StoryGPT-V}, which leverages the merits of the latent diffusion (LDM) and LLM to produce images with consistent and high-quality characters grounded on given story descriptions. First, we train a character-aware LDM, which takes character-augmented semantic embedding as input and includes the supervision of the cross-attention map using character segmentation masks, aiming to enhance character generation accuracy and faithfulness. In the second stage, we enable an alignment between the output of LLM and the character-augmented embedding residing in the input space of the first-stage model. This harnesses the reasoning ability of LLM to address ambiguous references and the comprehension capability to memorize the context. We conduct comprehensive experiments on two visual story visualization benchmarks. Our model reports superior quantitative results and consistently generates accurate characters of remarkable quality with low memory consumption. Our code is publicly available at: \href{https://xiaoqian-shen.github.io/StoryGPT-V}{https://xiaoqian-shen.github.io/StoryGPT-V}.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
