一种用于精确文档元素定位的大型多模态模型
An LMM for Precisely Grounding Elements in Documents
AI总结：
提出PreciseDoc，一种通过合成数据与强化学习联合训练实现文档元素精确定位的大型多模态模型，显著提升文档理解与视觉定位精度。
摘要：
文档中的视觉定位是大型多模态模型（LMM）在文档理解、深度研究和文档错误检测等领域的关键能力。然而，现有方法在文本丰富的文档图像中定位精度较差，常常无法准确找到可靠推理所需的关键文档元素。为了解决这一问题，我们引入了PreciseDoc，一种专门设计用于精确元素定位的LMM，并可进一步优化用于文档VQA任务。具体而言，为了增强基本定位能力，我们通过两条能够批量生产高质量文档及其细粒度坐标配对元数据的流程构建了具有挑战性的训练数据，包括具有相机效果的合成手填文档。该模型开发了超越简单单文本定位的更真实世界功能，例如从简历中定位个人信息。此外，我们引入了一种视觉接地推理的训练范式，其中接地和推理通过强化学习联合监督，以提高接地证据的贡献。在多个基准上的综合评估证明了所提出的数据和方法在文档空间定位和文档理解方面的优势。
英文摘要：
Visual grounding in documents is a crucial ability for Large Multimodal Models (LMMs) in areas such as document understanding, deep research and document error detection. However, existing approaches exhibit poor grounding precision in text-rich document images, often failing to accurately locate the critical document elements needed for reliable reasoning. To address this gap, we introduce PreciseDoc, an LMM specifically designed for precise element grounding and can be further optimized for Document VQA tasks. Specifically, to enhance the basic localization capability, we construct challenging training data by two pipelines capable of mass-producing high-quality documents with paired metadata of fine-grained coordinates, including synthetic hand-filled documents with camera effects. The model develops more real-world functions beyond straightforward localization of single text, such as locating personal information from CVs. Furthermore, we introduce a training paradigm for visual grounded reasoning where the grounding and reasoning are supervised jointly with reinforcement learning to improve the contribution of the grounded evidence. A comprehensive evaluation on various benchmarks demonstrates the advantage of the proposed data and methods in document spatial grounding and document understanding.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
