通用多语言文档编码器
A General-Purpose Multilingual Document Encoder
AI总结：
提出层次化多语言文档编码器HMDE，利用维基百科和跨语言对比学习预训练，在跨语言文档分类与检索任务上优于基线，并泛化至未见语言。
摘要：
大规模多语言预训练变换器（MMTs）极大地推动了多语言自然语言处理，特别是NLP模型的跨语言迁移领域的发展。虽然大量工作利用MMTs挖掘平行语料并生成双语文档嵌入，但致力于训练可用于监督和无监督文档级任务的通用（大规模）多语言文档编码器的努力相对较少。在本工作中，我们将大规模多语言文档编码器预训练为一种层次化变换器模型（HMDE），其中浅层文档变换器对由最先进的预训练多语言句子编码器产生的句子表示进行上下文化处理。我们利用维基百科作为现成的可比文档来源来创建训练数据，并通过跨语言对比目标训练HMDE，进一步利用维基百科的类别层次结构来构建困难负样本。我们在两个最普遍且突出的跨语言文档级任务中评估HMDE的有效性：（1）主题文档分类的跨语言迁移和（2）跨语言文档检索。HMDE显著优于（i）基于片段表示的聚合方法和（ii）多语言Longformer。关键在于，由于其大规模多语言的下层变换器，HMDE成功泛化到文档级预训练中未见过的语言。我们在https://github.com/ogaloglu/pre-training-multilingual-document-encoders 公开发布了我们的代码和模型。
英文摘要：
Massively multilingual pretrained transformers (MMTs) have tremendously pushed the state of the art on multilingual NLP and cross-lingual transfer of NLP models in particular. While a large body of work leveraged MMTs to mine parallel data and induce bilingual document embeddings, much less effort has been devoted to training general-purpose (massively) multilingual document encoder that can be used for both supervised and unsupervised document-level tasks. In this work, we pretrain a massively multilingual document encoder as a hierarchical transformer model (HMDE) in which a shallow document transformer contextualizes sentence representations produced by a state-of-the-art pretrained multilingual sentence encoder. We leverage Wikipedia as a readily available source of comparable documents for creating training data, and train HMDE by means of a cross-lingual contrastive objective, further exploiting the category hierarchy of Wikipedia for creation of difficult negatives. We evaluate the effectiveness of HMDE in two arguably most common and prominent cross-lingual document-level tasks: (1) cross-lingual transfer for topical document classification and (2) cross-lingual document retrieval. HMDE is significantly more effective than (i) aggregations of segment-based representations and (ii) multilingual Longformer. Crucially, owing to its massively multilingual lower transformer, HMDE successfully generalizes to languages unseen in document-level pretraining. We publicly release our code and models at https://github.com/ogaloglu/pre-training-multilingual-document-encoders .
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
