从分镜到散文：从漫画生成文学叙事
From Panels to Prose: Generating Literary Narratives from Comics
AI总结：
本文提出自动化系统Magiv3，结合大型视觉语言模型，从漫画生成文学叙事，以解决视障读者访问问题，并发布3300多个漫画分镜标注数据集。
摘要：
漫画长期以来一直是一种流行的叙事形式，通过视觉上引人入胜的画面吸引着全球观众。然而，漫画的视觉特性给视障读者带来了重大障碍，限制了他们对这些引人入胜的故事的访问。在这项工作中，我们为这一可访问性挑战提供了一个务实的解决方案，即开发一个自动化系统，从漫画中生成基于文本的文学叙事。我们的方法旨在创造一种唤起情感且沉浸式的散文，不仅传达原始叙事，还捕捉角色的深度和复杂性、他们之间的互动以及他们所处的生动场景。为此，我们做出了以下贡献：（1）我们提出了一个统一模型Magiv3，它在漫画理解的各种功能任务中表现出色，例如定位分镜、角色、文本和对话气泡尾部，执行OCR，角色接地等。（2）我们发布了超过3300个日本漫画分镜的人工标注标题，以及角色接地标注，并基准测试了大型视觉语言模型在理解漫画图像方面的能力。（3）最后，我们展示了如何将大型视觉语言模型与Magiv3集成，生成无缝的文学叙事，使视障受众能够参与漫画叙事的深度和丰富性。
英文摘要：
Comics have long been a popular form of storytelling, offering visually engaging narratives that captivate audiences worldwide. However, the visual nature of comics presents a significant barrier for visually impaired readers, limiting their access to these engaging stories. In this work, we provide a pragmatic solution to this accessibility challenge by developing an automated system that generates text-based literary narratives from manga comics. Our approach aims to create an evocative and immersive prose that not only conveys the original narrative but also captures the depth and complexity of characters, their interactions, and the vivid settings in which they reside. To this end we make the following contributions: (1) We present a unified model, Magiv3, that excels at various functional tasks pertaining to comic understanding, such as localising panels, characters, texts, and speech-bubble tails, performing OCR, grounding characters etc. (2) We release human-annotated captions for over 3300 Japanese comic panels, along with character grounding annotations, and benchmark large vision-language models in their ability to understand comic images. (3) Finally, we demonstrate how integrating large vision-language models with Magiv3, can generate seamless literary narratives that allows visually impaired audiences to engage with the depth and richness of comic storytelling.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
