面向文档图像机器翻译的基于多模态大语言模型的单模态到混合模态对齐
Single-to-mix Modality Alignment with Multimodal Large Language Model for Document Image Machine Translation
AI总结：
本文提出M4Doc框架，利用多模态大语言模型将仅图像编码器与多模态表示对齐，使轻量级模型学习视觉-文本关联，推理时绕过MLLM保持高效，显著提升跨域翻译质量。
摘要：
文档图像机器翻译（DIMT）旨在翻译文档图像中的文本，由于训练数据有限以及视觉与文本信息间复杂的相互作用，面临泛化挑战。为应对这些挑战，我们引入了M4Doc，这是一种利用多模态大语言模型（MLLM）的新型单模态到混合模态对齐框架。M4Doc将仅图像编码器与在大规模文档图像数据集上预训练的MLLM多模态表示进行对齐。这种对齐使轻量级DIMT模型能够在训练期间学习关键的视觉-文本关联。在推理期间，M4Doc绕过MLLM，在保持计算效率的同时受益于其多模态知识。综合实验表明，翻译质量有显著提升，特别是在跨域泛化和复杂文档图像场景中。
英文摘要：
Document Image Machine Translation (DIMT) aims to translate text within document images, facing generalization challenges due to limited training data and the complex interplay between visual and textual information. To address these challenges, we introduce M4Doc, a novel single-to-mix modality alignment framework leveraging Multimodal Large Language Models (MLLMs). M4Doc aligns an image-only encoder with the multimodal representations of an MLLM, pre-trained on large-scale document image datasets. This alignment enables a lightweight DIMT model to learn crucial visual-textual correlations during training. During inference, M4Doc bypasses the MLLM, maintaining computational efficiency while benefiting from its multimodal knowledge. Comprehensive experiments demonstrate substantial improvements in translation quality, especially in cross-domain generalization and challenging document image scenarios.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
