arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2007.06162cs.CL
收藏

你有合适的剪刀吗?基于蒙特卡罗方法的预训练语言模型裁剪

Do You Have the Right Scissors? Tailoring Pre-trained Language Models via Monte-Carlo Methods

ByteDance(字节跳动)

Ning Miao, Yuxuan Song, Hao Zhou, Lei Li

摘要:

在大型语料库上预训练语言模型并根据特定任务数据对其进行微调是一种常见的方法。在实践中,我们观察到在小数据集上微调预训练模型可能会导致高估和/或低估问题。在本文中,我们提出了MC-Tailor,一种新的方法,以减轻上述问题的文本生成任务的截断和转移的概率质量从高估的地区低估的。在多种文本生成数据集上的实验表明,MC-Tailor的性能始终显著优于微调方法。我们的代码可以在这个URL上找到。

英文摘要:

It has been a common approach to pre-train a language model on a large corpus and fine-tune it on task-specific data. In practice, we observe that fine-tuning a pre-trained model on a small dataset may lead to over- and/or under-estimation problem. In this paper, we propose MC-Tailor, a novel method to alleviate the above issue in text generation tasks by truncating and transferring the probability mass from over-estimated regions to under-estimated ones. Experiments on a variety of text generation datasets show that MC-Tailor consistently and significantly outperforms the fine-tuning approach. Our code is available at this url.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑