让讲故事生成生动故事：一种富有表现力且流畅的多模态讲故事系统
Let Storytelling Tell Vivid Stories: An Expressive and Fluent Multimodal Storyteller
AI总结：
针对现有多模态讲故事方法生成情节简单的问题，提出LLaMS流水线，结合序列数据自动增强、文本推理架构与SQ-Adapter模块，实现更具表现力和一致性的人类水平故事生成，性能超越现有SOTA方法。
摘要：
讲故事任务旨在基于有序图像流生成合理且生动的叙事。对图像故事主题的忠实度以及故事情节的发散性是吸引读者持续阅读的关键。以往的工作迭代式地改进了多模态的对齐效果，但最终导致为图像流生成的故事情节过于简单。在本研究中，我们提出了一种名为LLaMS的新流水线，用于生成兼具表现力与一致性的人类水平多模态故事。具体而言，通过充分挖掘大语言模型（LLM）中的常识知识，我们首先采用序列数据自动增强策略来提升事实内容的表达力，并利用文本推理架构实现富有表现力的故事生成与预测。其次，我们提出了用于故事插图生成的SQ-Adapter模块，该模块能够保持序列一致性。我们通过人工评估开展了数值实验，以验证所提出的LLaMS的优越性。评估结果表明，与以往的最优（SOTA）方法相比，LLaMS实现了当前最优的讲故事性能，取得了86%的相关性胜率与100%的一致性胜率。此外，我们还开展了消融实验，以验证所提出的序列数据增强方法与SQ-Adapter的有效性。
英文摘要：
Storytelling aims to generate reasonable and vivid narratives based on an ordered image stream. The fidelity to the image story theme and the divergence of story plots attract readers to keep reading. Previous works iteratively improved the alignment of multiple modalities but ultimately resulted in the generation of simplistic storylines for image streams. In this work, we propose a new pipeline, termed LLaMS, to generate multimodal human-level stories that are embodied in expressiveness and consistency. Specifically, by fully exploiting the commonsense knowledge within the LLM, we first employ a sequence data auto-enhancement strategy to enhance factual content expression and leverage a textual reasoning architecture for expressive story generation and prediction. Secondly, we propose SQ-Adatpter module for story illustration generation which can maintain sequence consistency. Numerical results are conducted through human evaluation to verify the superiority of proposed LLaMS. Evaluations show that LLaMS achieves state-of-the-art storytelling performance and 86% correlation and 100% consistency win rate as compared with previous SOTA methods. Furthermore, ablation experiments are conducted to verify the effectiveness of proposed sequence data enhancement and SQ-Adapter.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
