面向改进文档理解：基于多模态大语言模型的文本定位探索
Towards Improving Document Understanding: An Exploration on Text-Grounding via MLLMs
AI总结：
针对文档理解中文本定位能力挖掘不足的问题，提出TGDoc模型，通过构建含9.9万份演示文稿的数据集和1.2万条带位置信息的对话指令微调MLLMs，在多个文本密集基准上达SOTA性能。
摘要：
在文档理解领域，利用指令跟随数据微调多模态大语言模型（MLLMs）已取得显著进展。然而，文本密集场景下文本定位能力的潜力仍未得到充分挖掘。本文提出一种文本定位文档理解模型，命名为TGDoc，通过增强MLLMs识别图像中文本空间位置的能力来弥补这一不足。实验证据表明，文本定位可提升模型对文本内容的解读能力，从而提高其理解文本密集图像的熟练度。具体而言，我们整理了一个包含从互联网获取的9.9万份PowerPoint演示文稿的数据集。我们构建了包括文本检测、识别和定位在内的指令微调任务，以促进视觉编码器与大语言模型之间的一致性对齐。此外，我们收集了一组文本密集图像，并提示纯文本GPT-4生成1.2万条高质量对话，这些对话包含文本密集场景中的文本位置信息。通过将文本位置数据整合到指令中，TGDoc能够在视觉问答过程中熟练识别文本位置。大量实验表明，我们的方法在多个文本密集基准上取得了当前最优性能，验证了该方法的有效性。
英文摘要：
In the field of document understanding, significant advances have been made in the fine-tuning of Multimodal Large Language Models (MLLMs) with instruction-following data. Nevertheless, the potential of text-grounding capability within text-rich scenarios remains underexplored. In this paper, we present a text-grounding document understanding model, termed TGDoc, which addresses this deficiency by enhancing MLLMs with the ability to discern the spatial positioning of text within images. Empirical evidence suggests that text-grounding improves the model's interpretation of textual content, thereby elevating its proficiency in comprehending text-rich images. Specifically, we compile a dataset containing 99K PowerPoint presentations sourced from the internet. We formulate instruction tuning tasks including text detection, recognition, and spotting to facilitate the cohesive alignment between the visual encoder and large language model. Moreover, we curate a collection of text-rich images and prompt the text-only GPT-4 to generate 12K high-quality conversations, featuring textual locations within text-rich scenarios. By integrating text location data into the instructions, TGDoc is adept at discerning text locations during the visual question process. Extensive experiments demonstrate that our method achieves state-of-the-art performance across multiple text-rich benchmarks, validating the effectiveness of our method.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
