先看后说：通过视觉定位增强关键信息提取
See then Tell: Enhancing Key Information Extraction with Vision Grounding
AI总结：
针对传统 KIE 依赖 OCR 且现有图像到文本方法缺乏视觉定位的问题，提出带 <see> 标记和坐标解码的端到端 STNet，并构建 TVG 数据集，在多个基准上达到 SOTA。
摘要：
在数字时代，理解融合了文本、复杂版式和图像的视觉丰富文档的能力至关重要。传统的关键信息提取（KIE）方法主要依赖光学字符识别（OCR），这往往会带来显著的延迟、计算开销和错误。当前绕过 OCR 的先进图像到文本方法通常只生成纯文本输出，而没有相应的视觉定位。本文提出 STNet（See then Tell Net，先看后说网络），这是一种新颖的端到端模型，旨在提供精确答案以及相关视觉定位。STNet 的独特之处在于使用一个特殊的 <see> 标记来观察相关图像区域，并借助解码器解释与该标记关联的物理坐标。<see> 标记位于答案文本开头，使模型能够先“看”——观察图像中与输入问题相关的区域——然后“说”——给出清晰的文本回答。为增强模型的“看”能力，我们收集了大量结构化表格识别数据集。利用 GPT-4 先进的文本处理能力，我们构建了 TVG（TableQA with Vision Grounding，带视觉定位的表格问答）数据集，该数据集不仅提供基于文本的问答（QA）对，还为这些问答对加入精确视觉定位。我们的方法在 KIE 性能上取得显著提升，在 CORD、SROIE 和 DocVQA 等公开数据集上达到最先进结果。代码也将公开发布。
英文摘要：
In the digital era, the ability to understand visually rich documents that integrate text, complex layouts, and imagery is critical. Traditional Key Information Extraction (KIE) methods primarily rely on Optical Character Recognition (OCR), which often introduces significant latency, computational overhead, and errors. Current advanced image-to-text approaches, which bypass OCR, typically yield plain text outputs without corresponding vision grounding. In this paper, we introduce STNet (See then Tell Net), a novel end-to-end model designed to deliver precise answers with relevant vision grounding. Distinctively, STNet utilizes a unique <see> token to observe pertinent image areas, aided by a decoder that interprets physical coordinates linked to this token. Positioned at the outset of the answer text, the <see> token allows the model to first see-observing the regions of the image related to the input question-and then tell-providing articulated textual responses. To enhance the model's seeing capabilities, we collect extensive structured table recognition datasets. Leveraging the advanced text processing prowess of GPT-4, we develop the TVG (TableQA with Vision Grounding) dataset, which not only provides text-based Question Answering (QA) pairs but also incorporates precise vision grounding for these pairs. Our approach demonstrates substantial advancements in KIE performance, achieving state-of-the-art results on publicly available datasets such as CORD, SROIE, and DocVQA. The code will also be made publicly available.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
