arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2106.11539cs.CV
收藏

DocFormer:用于文档理解的端到端Transformer

DocFormer: End-to-End Transformer for Document Understanding

AWS AI(亚马逊云服务人工智能团队(AWS AI))

Srikar Appalaraju, Bhavan Jasani, Bhargava Urala Kota, Yusheng Xie, R. Manmatha

摘要:

我们提出了DocFormer --一个基于多模态Transformer的架构,用于可视化文档理解(VDU)任务。VDU是一个具有挑战性的问题,其目的是了解各种格式的文件(表格,收据等)。和布局。此外,DocFormer以无监督的方式进行预训练,使用精心设计的任务,鼓励多模式交互。DocFormer使用文本,视觉和空间特征,并使用新颖的多模态自我注意层将它们结合起来。DocFormer还跨模态共享学习到的空间嵌入,这使得模型可以轻松地将文本与视觉标记相关联,反之亦然。DocFormer在4个不同的数据集上进行评估,每个数据集都有很强的基线。DocFormer在所有这些方面都取得了最先进的结果,有时甚至比模型大4倍(参数数量)。

英文摘要:

We present DocFormer -- a multi-modal transformer based architecture for the task of Visual Document Understanding (VDU). VDU is a challenging problem which aims to understand documents in their varied formats (forms, receipts etc.) and layouts. In addition, DocFormer is pre-trained in an unsupervised fashion using carefully designed tasks which encourage multi-modal interaction. DocFormer uses text, vision and spatial features and combines them using a novel multi-modal self-attention layer. DocFormer also shares learned spatial embeddings across modalities which makes it easy for the model to correlate text to visual tokens and vice versa. DocFormer is evaluated on 4 different datasets each with strong baselines. DocFormer achieves state-of-the-art results on all of them, sometimes beating models 4x its size (in no. of parameters).

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑