arXiv 2505.054462025-05-08cs.CVcs.CL

面向上下文感知视觉文档理解的自适应标记语言生成

Adaptive Markup Language Generation for Contextually-Grounded Visual Document Understanding

CUHK MMLab（香港中文大学MML实验室）；vivo AI Lab（vivo人工智能实验室）；CPII under InnoHK（创新工场下的CPII）；Shanghai AI Lab & Shenzhen Institute of Advanced Technology, CAS（上海人工智能实验室及深圳先进技术研究所，中国科学院）

Han Xiao, Yina Xie, Guanxin Tan, Yinghao Chen, Rui Hu, Ke Wang, Aojun Zhou, Hao Li, Hao Shao, Xudong Lu, Peng Gao, Yafei Wen, Xiaoxin Chen, Shuai Ren, Hongsheng Li

备注：: CVPR2025

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出利用 Markdown、JSON、HTML 和 TiKZ 等标记语言自适应生成结构化文档表示，并构建 DocMark-Pile 与 DocMark-Instruct 两个细粒度数据集，显著提升视觉文档理解性能。

摘要：

随着富含文本的视觉内容不断增加，视觉文档理解变得至关重要。该领域面临重大挑战，因为需要有效整合视觉感知与文本理解，尤其是在布局复杂、类型多样的文档中。此外，现有用于该领域的微调数据集往往缺乏提供稳健理解所需的详细上下文信息，导致幻觉现象以及对视觉元素之间空间关系理解的局限性。为解决这些挑战，我们提出了一种创新流程，利用自适应生成标记语言（如 Markdown、JSON、HTML 和 TiKZ）来构建高度结构化的文档表示，并提供上下文感知的回答。我们引入了两个细粒度结构化数据集：DocMark-Pile，包含约 380 万对用于文档解析的预训练数据；以及 DocMark-Instruct，包含 62.4 万条用于基于上下文的指令跟随的微调数据标注。大量实验表明，我们提出的模型在一系列视觉文档理解基准上显著优于现有最先进的多模态大语言模型（MLLMs），在复杂视觉场景中增强了高级推理与理解能力。我们的代码和模型已发布于 https://github.com/Euphoria16/DocMark。

英文摘要：

Visual Document Understanding has become essential with the increase of text-rich visual content. This field poses significant challenges due to the need for effective integration of visual perception and textual comprehension, particularly across diverse document types with complex layouts. Moreover, existing fine-tuning datasets for this domain often fall short in providing the detailed contextual information for robust understanding, leading to hallucinations and limited comprehension of spatial relationships among visual elements. To address these challenges, we propose an innovative pipeline that utilizes adaptive generation of markup languages, such as Markdown, JSON, HTML, and TiKZ, to build highly structured document representations and deliver contextually-grounded responses. We introduce two fine-grained structured datasets: DocMark-Pile, comprising approximately 3.8M pretraining data pairs for document parsing, and DocMark-Instruct, featuring 624k fine-tuning data annotations for grounded instruction following. Extensive experiments demonstrate that our proposed model significantly outperforms existing state-of-theart MLLMs across a range of visual document understanding benchmarks, facilitating advanced reasoning and comprehension capabilities in complex visual scenarios. Our code and models are released at https://github. com/Euphoria16/DocMark.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。