StoryDALL-E：将预训练的文本到图像的Transformers用于故事延续
StoryDALL-E: Adapting Pretrained Text-to-Image Transformers for Story Continuation
摘要：
文本到图像合成的最新进展已经导致了大型预训练的transformer具有从给定文本生成可视化的出色能力。然而，这些模型并不适合像故事可视化这样的专门任务，这需要一个智能体在给定相应的标题序列的情况下产生一系列图像，形成一个叙事。此外，我们发现，故事可视化任务未能适应泛化到新的叙事中看不见的情节和人物。因此，我们首先提出了故事延续的任务，其中生成的视觉故事是有条件的源图像，允许更好地概括与新的字符的叙述。然后，我们用特定于任务的模块增强或“改造”预训练的文本到图像合成模型，用于（a）顺序图像生成和（b）从初始帧复制相关元素。然后，我们将探索预训练模型的全模型微调以及基于参数有效自适应的调整。我们评估我们的方法StoryDALL-E在两个现有的数据集，PororoSV和FlintstonesSV，并介绍了一个新的数据集DiDeMoSV从视频字幕数据集收集。我们还开发了一个基于生成对抗网络（GAN）的故事延续模型StoryGANc，并将其与StoryDALL-E模型进行比较，以展示我们方法的优势。我们表明，我们的改造方法在故事延续方面优于基于GAN的模型，并有助于从源图像中复制视觉元素，从而提高生成的视觉故事的连续性。最后，我们的分析表明，预先训练的变形金刚很难理解包含多个字符的叙述。总的来说，我们的工作表明，预训练的文本到图像合成模型可以适用于复杂和低资源的任务，如故事延续。
英文摘要：
Recent advances in text-to-image synthesis have led to large pretrained transformers with excellent capabilities to generate visualizations from a given text. However, these models are ill-suited for specialized tasks like story visualization, which requires an agent to produce a sequence of images given a corresponding sequence of captions, forming a narrative. Moreover, we find that the story visualization task fails to accommodate generalization to unseen plots and characters in new narratives. Hence, we first propose the task of story continuation, where the generated visual story is conditioned on a source image, allowing for better generalization to narratives with new characters. Then, we enhance or 'retro-fit' the pretrained text-to-image synthesis models with task-specific modules for (a) sequential image generation and (b) copying relevant elements from an initial frame. Then, we explore full-model finetuning, as well as prompt-based tuning for parameter-efficient adaptation, of the pre-trained model. We evaluate our approach StoryDALL-E on two existing datasets, PororoSV and FlintstonesSV, and introduce a new dataset DiDeMoSV collected from a video-captioning dataset. We also develop a model StoryGANc based on Generative Adversarial Networks (GAN) for story continuation, and compare it with the StoryDALL-E model to demonstrate the advantages of our approach. We show that our retro-fitting approach outperforms GAN-based models for story continuation and facilitates copying of visual elements from the source image, thereby improving continuity in the generated visual story. Finally, our analysis suggests that pretrained transformers struggle to comprehend narratives containing several characters. Overall, our work demonstrates that pretrained text-to-image synthesis models can be adapted for complex and low-resource tasks like story continuation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
