视觉丰富文档（如传单、横幅、杂志文章）是物理或数字文档，它们利用视觉线索来增强其语义。这些文档中包含的信息是临时性的，且往往不完整。现有支持对这些文档进行结构化查询的工作并未考虑到这一点。这使得对从这些文档查询到的信息进行语境化处理并从中收集可操作的见解变得困难。我们提出 Juno——一个跨模态实体匹配框架来解决这一局限。它通过将文档中的文本片段与外部数据库中语义相似的元组进行匹配，从而用补充信息增强异构文档。我们在这项工作中的主要贡献是一个带有注意力机制的深度神经网络，它超越了传统的基于关键词的匹配，在没有任何关于文档类型或底层模式的先验知识的情况下，通过在多模态编码空间上对齐文本片段和关系元组来找到匹配的元组。在多个真实世界数据集上的详尽实验表明，Juno 能够泛化到具有多样布局和格式的异构文档。它比最先进的基线方法高出超过 6 个 F1 分数点，同时所需的人工标注样本最多减少 60%。我们的实验进一步表明，Juno 是一个计算上稳健的框架。我们只需训练它一次，然后就可以动态地将其适配到多个资源受限的环境中，而不会牺牲其下游性能。这使得它适合在各种边缘设备上进行端侧部署。据我们所知，我们的工作是首个研究视觉丰富文档信息不完整性并提出一个可泛化、高性能且计算稳健的框架以端到端方式解决该问题的工作。
英文摘要：
Visually rich documents (e.g. leaflets, banners, magazine articles) are physical or digital documents that utilize visual cues to augment their semantics. Information contained in these documents are ad-hoc and often incomplete. Existing works that enable structured querying on these documents do not take this into account. This makes it difficult to contextualize the information retrieved from querying these documents and gather actionable insights from them. We propose Juno -- a cross-modal entity matching framework to address this limitation. It augments heterogeneous documents with supplementary information by matching a text span in the document with semantically similar tuples from an external database. Our main contribution in this is a deep neural network with attention that goes beyond traditional keyword-based matching and finds matching tuples by aligning text spans and relational tuples on a multimodal encoding space without any prior knowledge about the document type or the underlying schema. Exhaustive experiments on multiple real-world datasets show that Juno generalizes to heterogeneous documents with diverse layouts and formats. It outperforms state-of-the-art baselines by more than 6 F1 points with up to 60% less human-labeled samples. Our experiments further show that Juno is a computationally robust framework. We can train it only once, and then adapt it dynamically for multiple resource-constrained environments without sacrificing its downstream performance. This makes it suitable for on-device deployment in various edge-devices. To the best of our knowledge, ours is the first work that investigates the information incompleteness of visually rich documents and proposes a generalizable, performant and computationally robust framework to address it in an end-to-end way.
WordScape: a Pipeline to extract multilingual, visually rich Documents with Layout Annotations from Web Crawl Data
ETH Zurich（苏黎世联邦理工学院）；University of Chicago（芝加哥大学）；INSAIT, Sofia University（索非亚大学INSAIT学院）；TU darmstadt（达姆施塔特工业大学）；Argonne National Laboratory（阿贡国家实验室）；UIUC（伊利诺伊大学厄巴纳-香槟分校）
