VIST-GPT：借助 LLM 开启视觉叙事时代？
VIST-GPT: Ushering in the Era of Visual Storytelling with LLMs?
AI总结：
本文提出 VIST-GPT，通过改造基于 transformer 的大型多模态模型并利用 VIST 数据集生成视觉连贯叙事，同时采用 RoViST 与 GROOVIST 无参考指标实现更贴近人类判断的评估。
摘要：
视觉叙事是一个结合计算机视觉与自然语言处理的交叉领域，旨在从图像序列生成连贯叙述。本文提出一种新方法，利用多模态模型的最新进展，特别是改造基于 transformer 的架构和大型多模态模型，以完成视觉叙事任务。借助大规模 Visual Storytelling（VIST）数据集，我们的 VIST-GPT 模型能够生成具有视觉依据且符合上下文的叙述。我们指出 BLEU、METEOR、ROUGE 和 CIDEr 等传统评估指标的局限，它们并不适合该任务。因此，我们采用 RoViST 和 GROOVIST，这两种为评估视觉叙事设计的新型无参考指标关注视觉依据、连贯性和非冗余性。这些指标能够更细致地评估叙述质量，并与人类判断密切一致。
英文摘要：
Visual storytelling is an interdisciplinary field combining computer vision and natural language processing to generate cohesive narratives from sequences of images. This paper presents a novel approach that leverages recent advancements in multimodal models, specifically adapting transformer-based architectures and large multimodal models, for the visual storytelling task. Leveraging the large-scale Visual Storytelling (VIST) dataset, our VIST-GPT model produces visually grounded, contextually appropriate narratives. We address the limitations of traditional evaluation metrics, such as BLEU, METEOR, ROUGE, and CIDEr, which are not suitable for this task. Instead, we utilize RoViST and GROOVIST, novel reference-free metrics designed to assess visual storytelling, focusing on visual grounding, coherence, and non-redundancy. These metrics provide a more nuanced evaluation of narrative quality, aligning closely with human judgment.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
