arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2306.01733cs.CVcs.CLcs.LG
收藏

DocFormerv2:用于文档理解的局部特征

DocFormerv2: Local Features for Document Understanding

AWS AI Labs(AWS人工智能实验室);School of Computing at University of Utah(犹他大学计算学院)

Srikar Appalaraju, Peng Tang, Qi Dong, Nishant Sankaran, Yichu Zhou, R. Manmatha

AI总结:

本文提出多模态编码器-解码器 transformer DocFormerv2,通过非对称无监督预训练促进视觉、语言与空间特征的局部对齐,并在九个文档数据集及场景文本 VQA 任务上取得先进结果。

摘要:

我们提出 DocFormerv2,这是一种用于视觉文档理解(Visual Document Understanding,VDU)的多模态 transformer。VDU 领域需要理解文档(而不仅仅是 OCR 预测),例如从表单中提取信息、文档 VQA 以及其他任务。VDU 具有挑战性,因为它要求模型理解多种模态(视觉、语言和空间)以作出预测。我们的方法称为 DocFormerv2,是一种编码器-解码器 transformer,将视觉、语言和空间特征作为输入。DocFormerv2 使用以非对称方式采用的无监督任务进行预训练,即在编码器上使用两个新颖的文档任务,在自回归解码器上使用一个任务。这些无监督任务经过精心设计,以确保预训练能够促进多种模态之间的局部特征对齐。DocFormerv2 在九个数据集上接受评估时,相较强基线展现出最先进性能,例如 TabFact(4.3%)、InfoVQA(1.4%)、FUNSD(1%)。此外,为展示泛化能力,在三个涉及场景文本的 VQA 任务上,DocFormerv2 优于以往规模相当的模型,甚至在某些任务上表现优于大得多的模型(如 GIT2、PaLi 和 Flamingo)。大量消融实验表明,得益于其预训练,DocFormerv2 比 VDU 中的现有方法更好地理解多种模态。

英文摘要:

We propose DocFormerv2, a multi-modal transformer for Visual Document Understanding (VDU). The VDU domain entails understanding documents (beyond mere OCR predictions) e.g., extracting information from a form, VQA for documents and other tasks. VDU is challenging as it needs a model to make sense of multiple modalities (visual, language and spatial) to make a prediction. Our approach, termed DocFormerv2 is an encoder-decoder transformer which takes as input - vision, language and spatial features. DocFormerv2 is pre-trained with unsupervised tasks employed asymmetrically i.e., two novel document tasks on encoder and one on the auto-regressive decoder. The unsupervised tasks have been carefully designed to ensure that the pre-training encourages local-feature alignment between multiple modalities. DocFormerv2 when evaluated on nine datasets shows state-of-the-art performance over strong baselines e.g. TabFact (4.3%), InfoVQA (1.4%), FUNSD (1%). Furthermore, to show generalization capabilities, on three VQA tasks involving scene-text, Doc- Formerv2 outperforms previous comparably-sized models and even does better than much larger models (such as GIT2, PaLi and Flamingo) on some tasks. Extensive ablations show that due to its pre-training, DocFormerv2 understands multiple modalities better than prior-art in VDU.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑