arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2107.14596cs.CL
收藏

基于简化多模态预训练模型的多阶段预训练

Multi-stage Pre-training over Simplified Multimodal Pre-training Models

Beijing University of Posts and Telecommunications(北京邮电大学)

Tongtong Liu, Fangxiang Feng, Xiaojie Wang

摘要:

LXMERT等多模态预训练模型在下游任务中取得了优异的效果。然而,当前的预训练模型需要大量的训练数据,并且具有巨大的模型大小,这使得它们难以应用于低资源情况。如何在预训练数据较少、模型规模较小的前提下,获得与较大模型相似甚至更好的性能,成为一个重要的问题。在本文中,我们提出了一种新的多阶段预训练(MSP)方法,该方法使用文本和图像中从单词,短语到句子的不同粒度的信息来分阶段预训练模型。为了有效地从有限的语料库中获取多样化的知识,我们还设计了几个不同的预训练任务,以适应不同阶段的信息粒度。我们以简化的LXMERT(LXMERT-S)模型作为MSP方法的实验平台,该模型的参数仅为原始LXMERT模型的45.9%,训练数据仅为原始预训练数据的11.76%。实验结果表明,该方法在所有下游任务中的性能与原始LXMERT模型相当,甚至在图像-文本检索任务中优于原始模型。

英文摘要:

Multimodal pre-training models, such as LXMERT, have achieved excellent results in downstream tasks. However, current pre-trained models require large amounts of training data and have huge model sizes, which make them difficult to apply in low-resource situations. How to obtain similar or even better performance than a larger model under the premise of less pre-training data and smaller model size has become an important problem. In this paper, we propose a new Multi-stage Pre-training (MSP) method, which uses information at different granularities from word, phrase to sentence in both texts and images to pre-train the model in stages. We also design several different pre-training tasks suitable for the information granularity in different stage in order to efficiently capture the diverse knowledge from a limited corpus. We take a Simplified LXMERT (LXMERT- S), which has only 45.9% parameters of the original LXMERT model and 11.76% of the original pre-training data as the testbed of our MSP method. Experimental results show that our method achieves comparable performance to the original LXMERT model in all downstream tasks, and even outperforms the original model in Image-Text Retrieval task.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑