Self-supervised pre-training techniques have achieved remarkable progress in Document AI. Most multimodal pre-trained models use a masked language modeling objective to learn bidirectional representations on the text modality, but they differ in pre-training objectives for the image modality. This discrepancy adds difficulty to multimodal representation learning. In this paper, we propose \textbf{LayoutLMv3} to pre-train multimodal Transformers for Document AI with unified text and image masking. Additionally, LayoutLMv3 is pre-trained with a word-patch alignment objective to learn cross-modal alignment by predicting whether the corresponding image patch of a text word is masked. The simple unified architecture and training objectives make LayoutLMv3 a general-purpose pre-trained model for both text-centric and image-centric Document AI tasks. Experimental results show that LayoutLMv3 achieves state-of-the-art performance not only in text-centric tasks, including form understanding, receipt understanding, and document visual question answering, but also in image-centric tasks such as document image classification and document layout analysis. The code and models are publicly available at \url{https://aka.ms/layoutlmv3}.
Out-of-Distribution Detection with Attention Head Masking for Multimodal Document Classification
University of Bristol（布里斯托尔大学）；Amazon GenAI（亚马逊生成式人工智能部门）；James Silberrad Brown Center for Artificial Intelligence（詹姆斯·西尔伯拉德·布朗人工智能中心）；Carnegie Mellon University（卡内基梅隆大学）；Stanford University（斯坦福大学）
Domain adaptation of Russian ModernBERT for long legal documents
Independent Researcher（独立研究者）；National Research Nuclear University MEPhI（国立核研究大学 MEPhI）；Moscow Center for Advanced Studies（莫斯科高级研究中心）；University of Waterloo（滑铁卢大学）
Beyond Image-Text Matching: Verb Understanding in Multimodal Transformers Using Guided Masking
Faculty of Information Technology, Brno University of Technology（布尔诺理工大学信息技术学院）；Kempelen Institute of Intelligent Technologies（肯佩伦智能技术研究所）；George Mason University（乔治梅森大学）
PMI-Masking: Principled masking of correlated spans
AI21 Labs（AI21实验室）
随机均匀地掩蔽标记构成了诸如BERT之类的掩蔽语言模型（Masked Language Models，MLMs）的预训练中的常见缺陷。我们表明，这种均匀的掩蔽允许MLM通过锁定浅本地信号来最小化其训练目标，从而导致预训练效率低下和次优下游性能。为了解决这一缺陷，我们提出了PMI-Masking，一个原则性的掩蔽策略的基础上的概念逐点互信息（PMI），共同掩盖一个令牌n元语法，如果它表现出高搭配的语料库。PMI掩蔽激励、统一和改进了先前的更启发式的方法，这些方法试图解决随机均匀标记掩蔽的缺点，例如全词掩蔽、实体/短语掩蔽和随机跨度掩蔽。具体来说，我们通过实验表明，PMI掩蔽在一半的训练时间内达到了先前掩蔽方法的性能，并且在训练结束时始终提高了性能。
Retrieval Oriented Masking Pre-training Language Model for Dense Passage Retrieval
Alibaba Group（阿里巴巴集团）
预训练的语言模型（PTM）已被证明产生强大的文本表示密集段落检索任务。Masked Language Modeling（MLM）是预训练过程的一个主要子任务。然而，我们发现，传统的随机掩蔽策略倾向于选择大量的令牌，对段落检索任务的影响有限（例如，停止词和标点符号）。通过注意到术语重要性权重可以为段落检索提供有价值的信息，我们在此提出了替代的面向检索的掩蔽（称为ROM）策略，其中更重要的标记将具有更高的被掩蔽的概率，以捕获这种简单而重要的信息，以促进语言模型预训练过程。值得注意的是，所提出的新的令牌掩蔽方法不会改变原始PTM的架构和学习目标。我们的实验验证了所提出的ROM使术语重要性信息，以帮助语言模型预训练，从而实现更好的性能多通道检索基准。
