用于视觉文档理解的统一模态掩码序列到序列预训练
Sequence-to-Sequence Pre-training with Unified Modality Masking for Visual Document Understanding
AI总结：
提出GenDoc，一个基于编码器-解码器架构的序列到序列文档理解模型，通过统一模态掩码预训练和混合模态专家策略，在多种下游任务上取得优异性能，并在OCR质量不佳时表现出更强鲁棒性。
摘要：
本文提出了GenDoc，一个通用的序列到序列文档理解模型，通过在文本、图像和布局三种模态上进行统一掩码预训练。该模型采用编码器-解码器架构，与文档理解中常用的仅编码器模型相比，能够更好地适应具有多样输出格式的各种下游任务。除了以往编码器-解码器模型使用的传统文本填充任务外，我们的预训练还扩展到掩码图像标记预测和掩码布局预测任务。我们还设计了特定模态的指令，并采用解耦注意力和混合模态专家策略，以有效捕获每种模态所利用的信息。通过在文档理解的多个下游任务上进行大量实验评估，结果表明该模型与最先进方法相比，能够取得更优或具有竞争力的性能。我们的分析进一步表明，在OCR质量不完美的场景下，GenDoc比仅编码器模型更具鲁棒性。
英文摘要：
This paper presents GenDoc, a general sequence-to-sequence document understanding model pre-trained with unified masking across three modalities: text, image, and layout. The proposed model utilizes an encoder-decoder architecture, which allows for increased adaptability to a wide range of downstream tasks with diverse output formats, in contrast to the encoder-only models commonly employed in document understanding. In addition to the traditional text infilling task used in previous encoder-decoder models, our pre-training extends to include tasks of masked image token prediction and masked layout prediction. We also design modality-specific instruction and adopt both disentangled attention and the mixture-of-modality-experts strategy to effectively capture the information leveraged by each modality. Evaluation of the proposed model through extensive experiments on several downstream tasks in document understanding demonstrates its ability to achieve superior or competitive performance compared to state-of-the-art approaches. Our analysis further suggests that GenDoc is more robust than the encoder-only models in scenarios where the OCR quality is imperfect.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
