MaRU：连接视觉与语言的漫画检索与理解系统
MaRU: A Manga Retrieval and Understanding System Connecting Vision and Language
AI总结：
针对漫画分格结构带来的内容检索难题，提出多阶段多模态系统MaRU，集成目标检测、文本识别与多模态编码模块，实现高效的对话与场景检索，在相关任务上表现优异。
摘要：
漫画（Manga）是一种广受赞誉的日本漫画艺术形式，以其多样化的叙事和独特的艺术风格闻名。然而，漫画本身具有视觉化且结构复杂的特点，单张图像包含多个分格，这给内容检索带来了重大挑战。为解决这一问题，我们提出MaRU（Manga Retrieval and Understanding，漫画检索与理解），这是一个连接视觉与语言的多阶段系统，可高效检索漫画画格内的对话与场景。MaRU的架构集成了用于识别文本和画格边界框的目标检测模型、用于文本识别的视觉编解码模型、用于嵌入文本的文本编码器，以及将文本与视觉信息融合到统一嵌入空间以实现场景检索的视觉-文本编码器。严格评估表明，MaRU在端到端对话检索中表现出色，在场景检索方面也展现出可观的效果。
英文摘要：
Manga, a widely celebrated Japanese comic art form, is renowned for its diverse narratives and distinct artistic styles. However, the inherently visual and intricate structure of Manga, which comprises images housing multiple panels, poses significant challenges for content retrieval. To address this, we present MaRU (Manga Retrieval and Understanding), a multi-staged system that connects vision and language to facilitate efficient search of both dialogues and scenes within Manga frames. The architecture of MaRU integrates an object detection model for identifying text and frame bounding boxes, a Vision Encoder-Decoder model for text recognition, a text encoder for embedding text, and a vision-text encoder that merges textual and visual information into a unified embedding space for scene retrieval. Rigorous evaluations reveal that MaRU excels in end-to-end dialogue retrieval and exhibits promising results for scene retrieval.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
