文档布局分析（DLA）旨在将文档图像分解为高级语义区域（即图形、表格、文本和背景）。由于文档对象在布局、大小、宽高比、纹理等方面的多样性，创建具有强泛化能力的 DLA 框架是一项挑战。许多研究人员通过合成数据来构建大型训练集以应对这一挑战。然而，合成的训练数据风格各异且质量不稳定。此外，源数据与目标数据之间存在较大差距。在本文中，我们提出了一种基于文档风格指导的无监督跨域 DLA 框架。我们将文档质量评估和文档跨域分析集成到一个统一的框架中。我们的框架由三个组件组成：Document Layout Generator (GLD)、Document Elements Decorator (GED) 和 Document Style Discriminator (DSD)。GLD 用于生成文档布局，GED 用于填充文档布局元素，DSD 用于文档质量评估和跨域指导。首先，我们应用 GLD 预测生成文档的位置。然后，我们设计了一种基于美学指导的新算法来填充文档位置。最后，我们使用对比学习来评估文档的质量。此外，我们设计了一种新策略，将文档质量评估组件转变为文档跨域风格指导组件。我们的框架是一个无监督的文档布局分析框架。我们通过大量实验证明，我们提出的方法取得了显著的性能。
英文摘要：
The document layout analysis (DLA) aims to decompose document images into high-level semantic areas (i.e., figures, tables, texts, and background). Creating a DLA framework with strong generalization capabilities is a challenge due to document objects are diversity in layout, size, aspect ratio, texture, etc. Many researchers devoted this challenge by synthesizing data to build large training sets. However, the synthetic training data has different styles and erratic quality. Besides, there is a large gap between the source data and the target data. In this paper, we propose an unsupervised cross-domain DLA framework based on document style guidance. We integrated the document quality assessment and the document cross-domain analysis into a unified framework. Our framework is composed of three components, Document Layout Generator (GLD), Document Elements Decorator(GED), and Document Style Discriminator(DSD). The GLD is used to document layout generates, the GED is used to document layout elements fill, and the DSD is used to document quality assessment and cross-domain guidance. First, we apply GLD to predict the positions of the generated document. Then, we design a novel algorithm based on aesthetic guidance to fill the document positions. Finally, we use contrastive learning to evaluate the quality assessment of the document. Besides, we design a new strategy to change the document quality assessment component into a document cross-domain style guide component. Our framework is an unsupervised document layout analysis framework. We have proved through numerous experiments that our proposed method has achieved remarkable performance.
Fudan University（复旦大学）；East China Normal University（华东师范大学）；Videt Lab（Videt实验室）；Zhongshan Hospital, Fudan University（复旦大学附属中山医院）
Document layout analysis (DLA) plays an important role in information extraction and document understanding. At present, document layout analysis has reached a milestone achievement, however, document layout analysis of non-Manhattan is still a challenge. In this paper, we propose an image layer modeling method to tackle this challenge. To measure the proposed image layer modeling method, we propose a manually-labeled non-Manhattan layout fine-grained segmentation dataset named FPD. As far as we know, FPD is the first manually-labeled non-Manhattan layout fine-grained segmentation dataset. To …
PRHLT Research Center（PRHLT研究中心）；Universitat Politècnica de València（瓦伦西亚理工大学）
Document Layout Analysis is a fundamental step in Handwritten Text Processing systems, from the extraction of the text lines to the type of zone it belongs to. We present a system based on artificial neural networks which is able to determine not only the baselines of text lines present in the document, but also performs geometric and logic layout analysis of the document. Experiments in three different datasets demonstrate the potential of the method and show competitive results with respect to state-of-the-art methods.
PRHLT Research Center（PRHLT研究中心）；Universitat Politècnica de València（瓦伦西亚理工大学）
Automatically recognizing the layout of handwritten documents is an important step towards useful extraction of information from those documents. The most common application is to feed downstream applications such as automatic text recognition and keyword spotting; however, the recognition of the layout also helps to establish relationships between elements in the document which allows to enrich the information that can be extracted. Most of the modern document layout analysis systems are designed to address only one part of the document layout problem, namely: baseline detection or region seg…
Hikvision Research Institute（海康威视研究院）；Zhejiang University（浙江大学）
Document layout analysis is crucial for understanding document structures. On this task, vision and semantics of documents, and relations between layout components contribute to the understanding process. Though many works have been proposed to exploit the above information, they show unsatisfactory results. NLP-based methods model layout analysis as a sequence labeling task and show insufficient capabilities in layout modeling. CV-based methods model layout analysis as a detection or segmentation task, but bear limitations of inefficient modality fusion and lack of relation modeling between l…
Extraction of text regions and individual text lines from historic documents is necessary for automatic transcription. We propose extending a CNN-based text baseline detection system by adding line height and text block boundary predictions to the model output, allowing the system to extract more comprehensive layout information. We also show that pixel-wise text orientation prediction can be used for processing documents with multiple text orientations. We demonstrate that the proposed method performs well on the cBAD baseline detection dataset. Additionally, we benchmark the method on newly …
M$^{6}$Doc: A Large-Scale Multi-Format, Multi-Type, Multi-Layout, Multi-Language, Multi-Annotation Category Dataset for Modern Document Layout Analysis
South China University of Technology（华南理工大学）；Huawei Cloud Computing Technologies Co., Ltd.（华为云计算技术有限公司）；IntSig Information Co., Ltd.（合合信息科技股份有限公司）
Digitization of newspapers is of interest for many reasons including preservation of history, accessibility and search ability, etc. While digitization of documents such as scientific articles and magazines is prevalent in literature, one of the main challenges for digitization of newspaper lies in its complex layout (e.g. articles spanning multiple columns, text interrupted by images) analysis, which is necessary to preserve human read-order. This work provides a major breakthrough in the digitization of newspapers on three fronts: first, releasing a dataset of 3000 fully-annotated, real-worl…
Deep learning-based approaches for automatic document layout analysis and content extraction have the potential to unlock rich information trapped in historical documents on a large scale. One major hurdle is the lack of large datasets for training robust models. In particular, little training data exist for Asian languages. To this end, we present HJDataset, a Large Dataset of Historical Japanese Documents with Complex Layouts. It contains over 250,000 layout element annotations of seven types. In addition to bounding boxes and masks of the content regions, it also includes the hierarchical s…
South China University of Technology（华南理工大学）；Alibaba Group（阿里巴巴集团）
In this paper, we propose an end-to-end trainable framework for restoring historical documents content that follows the correct reading order. In this framework, two branches named character branch and layout branch are added behind the feature extraction network. The character branch localizes individual characters in a document image and recognizes them simultaneously. Then we adopt a post-processing method to group them into text lines. The layout branch based on fully convolutional network outputs a binary mask. We then use Hough transform for line detection on the binary mask and combine …
This paper presents an open tool for standardizing the evaluation process of the layout analysis task of document images at pixel level. We introduce a new evaluation tool that is both available as a standalone Java application and as a RESTful web service. This evaluation tool is free and open-source in order to be a common tool that anyone can use and contribute to. It aims at providing as many metrics as possible to investigate layout analysis predictions, and also provide an easy way of visualizing the results. This tool evaluates document segmentation at pixel level, and support multi-lab…
East China Normal University（华东师范大学）；Videt Tech Ltd.（维迪特科技有限公司）
The document layout analysis (DLA) aims to split the document image into different interest regions and understand the role of each region, which has wide application such as optical character recognition (OCR) systems and document retrieval. However, it is a challenge to build a DLA system because the training data is very limited and lacks an efficient model. In this paper, we propose an end-to-end united network named Dynamic Residual Fusion Network (DRFN) for the DLA task. Specifically, we design a dynamic residual feature fusion module which can fully utilize low-dimensional information a…
Diachronic Document Dataset for Semantic Layout Analysis
Inria（法国国家信息与自动化研究所）；Paris Sciences & Lettres（巴黎文理研究大学）；Center for Digital Humanities and Multilateralism（数字人文与多边主义中心）；Geneva Graduate Institute（日内瓦高等研究院）；University of Geneva（日内瓦大学）
The University of Sydney（悉尼大学）；The University of Western Australia（西澳大学）
Recognizing the layout of unstructured digital documents is crucial when parsing the documents into the structured, machine-readable format for downstream applications. Recent studies in Document Layout Analysis usually rely on computer vision models to understand documents while ignoring other information, such as context information or relation of document components, which are vital to capture. Our Doc-GCN presents an effective way to harmonize and integrate heterogeneous aspects for Document Layout Analysis. We first construct graphs to explicitly describe four main aspects, including synt…
PARL: Position-Aware Relation Learning Network for Document Layout Analysis
Unisound AI Technology Co.Ltd（Unisound人工智能技术有限公司）；MAIS, Institute of Automation, CAS（MAIS，自动化研究所，中国科学院）；Beihang University（北航）；School of Computer Science and Technology, Tiangong University（天工大学计算机科学与技术学院）
Accurate document layout analysis is a key requirement for high-quality PDF document conversion. With the recent availability of public, large ground-truth datasets such as PubLayNet and DocBank, deep-learning models have proven to be very effective at layout detection and segmentation. While these datasets are of adequate size to train such models, they severely lack in layout variability since they are sourced from scientific article repositories such as PubMed and arXiv only. Consequently, the accuracy of the layout segmentation drops significantly when these models are applied on more chal…
Recently, various multimodal networks for Visually-Rich Document Understanding(VRDU) have been proposed, showing the promotion of transformers by integrating visual and layout information with the text embeddings. However, most existing approaches utilize the position embeddings to incorporate the sequence information, neglecting the noisy improper reading order obtained by OCR tools. In this paper, we propose a robust layout-aware multimodal network named XYLayoutLM to capture and leverage rich layout information from proper reading orders produced by our Augmented XY Cut. Moreover, a Dilated…
East China Normal University（华东师范大学）；Shanghai Key Laboratory of Multidimensional Information Processing（上海市多维度信息处理重点实验室）
Document layout analysis (DLA) aims to divide a document image into different types of regions. DLA plays an important role in the document content understanding and information extraction systems. Exploring a method that can use less data for effective training contributes to the development of DLA. We consider a Human-in-the-loop (HITL) collaborative intelligence in the DLA. Our approach was inspired by the fact that the HITL push the model to learn from the unknown problems by adding a small amount of data based on knowledge. The HITL select key samples by using confidence. However, using c…
Parser-Oriented Structural Refinement for a Stable Layout Interface in Document Parsing
Unisound AI Technology Co., Ltd.（云知声智能科技股份有限公司）；School of Computer Science and Technology, Tianjin University（天津大学计算机科学与技术学院）；MAIS, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所多模态人工智能系统实验室）
A large amount of document data exists in unstructured form such as raw images without any text information. Designing a practical document image analysis system is a meaningful but challenging task. In previous work, we proposed an intelligent document analysis system PP-Structure. In order to further upgrade the function and performance of PP-Structure, we propose PP-StructureV2 in this work, which contains two subsystems: Layout Information Extraction and Key Information Extraction. Firstly, we integrate Image Direction Correction module and Layout Restoration module to enhance the function…
LLM-Guided Probabilistic Fusion for Label-Efficient Document Layout Analysis
Department of Computer Science, Iowa State University（计算机科学系，爱荷华州立大学）；Department of Civil, Construction & Environmental Engineering, Iowa State University（土木、建设与环境工程系，爱荷华州立大学）
University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）；Microsoft Azure Cognitive Services Research（微软Azure认知服务研究院）；Microsoft Azure Visual Document Intelligence（微软Azure视觉文档智能部门）
We propose Universal Document Processing (UDOP), a foundation Document AI model which unifies text, image, and layout modalities together with varied task formats, including document understanding and generation. UDOP leverages the spatial correlation between textual content and document image to model image, text, and layout modalities with one uniform representation. With a novel Vision-Text-Layout Transformer, UDOP unifies pretraining and multi-domain downstream tasks into a prompt-based sequence generation scheme. UDOP is pretrained on both large-scale unlabeled document corpora using inno…
University of California, Los Angeles（加州大学洛杉矶分校）；Google Research（谷歌研究院）
Document layout comprises both structural and visual (eg. font-sizes) information that is vital but often ignored by machine learning models. The few existing models which do use layout information only consider textual contents, and overlook the existence of contents in other modalities such as images. Additionally, spatial interactions of presented contents in a layout were never really fully exploited. To bridge this gap, we parse a document into content blocks (eg. text, table, image) and propose a novel layout-aware multimodal hierarchical framework, LAMPreT, to model the blocks and the w…
Users of OCR systems, from different institutions and scientific disciplines, prefer and produce different transcription styles. This presents a problem for training of consistent text recognition neural networks on real-world data. We propose to extend existing text recognition networks with a Transcription Style Block (TSB) which can learn from data to switch between multiple transcription styles without any explicit knowledge of transcription rules. TSB is an adaptive instance normalization conditioned by identifiers representing consistently transcribed documents (e.g. single document, doc…
Page layout analysis is a fundamental step in document processing which enables to segment a page into regions of interest. With highly complex layouts and mixed scripts, scholarly commentaries are text-heavy documents which remain challenging for state-of-the-art models. Their layout considerably varies across editions and their most important regions are mainly defined by semantic rather than graphical characteristics such as position or appearance. This setting calls for a comparison between textual, visual and hybrid approaches. We therefore assess the performances of two transformers (Lay…
