mPLUG-DocOwl：面向文档理解的模块化多模态大语言模型
mPLUG-DocOwl: Modularized Multimodal Large Language Model for Document Understanding
AI总结：
本文提出mPLUG-DocOwl，通过构建多任务指令微调数据集并联合训练纯语言、通用视觉-语言和文档任务，增强无OCR文档理解能力，并构建LLMDoc评测集，实验证明其优于现有多模态模型且泛化良好。
摘要：
文档理解是指从网页等各种数字文档中自动提取、分析和理解信息。现有的多模态大语言模型（MLLMs），包括mPLUG-Owl，已在浅层无OCR文本识别中展现出有前景的零样本能力，表明其具备无OCR文档理解的潜力。然而，若缺乏领域内训练，这些模型往往会忽略细粒度的OCR特征，例如复杂表格或大段文本，而这些特征对于无OCR文档理解至关重要。本文基于mPLUG-Owl提出mPLUG-DocOwl，用于无OCR文档理解。具体而言，我们首先构建了一个涵盖广泛视觉-文本理解任务的指令微调数据集。随后，通过统一的指令微调策略，在纯语言、通用视觉-语言和文档指令微调数据集上联合训练模型，以增强其无OCR文档理解能力。我们还构建了一个无OCR文档指令理解评测集LLMDoc，以更好地比较模型在指令遵循和文档理解方面的能力。实验结果表明，我们的模型优于现有多模态模型，展现出强大的文档理解能力。此外，在未进行特定微调的情况下，mPLUG-DocOwl在各种下游任务上具有良好的泛化性能。我们的代码、模型、训练数据和评测集可在https://github.com/X-PLUG/mPLUG-DocOwl获取。
英文摘要：
Document understanding refers to automatically extract, analyze and comprehend information from various types of digital documents, such as a web page. Existing Multi-model Large Language Models (MLLMs), including mPLUG-Owl, have demonstrated promising zero-shot capabilities in shallow OCR-free text recognition, indicating their potential for OCR-free document understanding. Nevertheless, without in-domain training, these models tend to ignore fine-grained OCR features, such as sophisticated tables or large blocks of text, which are essential for OCR-free document understanding. In this paper, we propose mPLUG-DocOwl based on mPLUG-Owl for OCR-free document understanding. Specifically, we first construct a instruction tuning dataset featuring a wide range of visual-text understanding tasks. Then, we strengthen the OCR-free document understanding ability by jointly train the model on language-only, general vision-and-language, and document instruction tuning dataset with our unified instruction tuning strategy. We also build an OCR-free document instruction understanding evaluation set LLMDoc to better compare models' capabilities on instruct compliance and document understanding. Experimental results show that our model outperforms existing multi-modal models, demonstrating its strong ability of document understanding. Besides, without specific fine-tuning, mPLUG-DocOwl generalizes well on various downstream tasks. Our code, models, training data and evaluation set are available at https://github.com/X-PLUG/mPLUG-DocOwl.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
