arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2210.05883cs.CL
收藏

AD-DROP:基于属性驱动的健壮语言模型微调

AD-DROP: Attribution-Driven Dropout for Robust Language Model Fine-Tuning

Sun Yat-sen University(中山大学);Meta AI

Tao Yang, Jinghao Deng, Xiaojun Quan, Qifan Wang, Shaoliang Nie

摘要:

当训练数据有限时,在下游任务上微调大的预训练语言模型容易遭受过拟合。虽然辍学被证明是一种有效的解毒剂,可以通过随机放弃一定比例的单位来解决问题,但现有的研究尚未研究其对自我注意力机制的影响。在本文中,我们通过自我注意归因来研究这个问题,发现降低低归因分数的注意位置可以加速训练并增加过拟合的风险。基于这一观察结果,我们提出了归因驱动的丢弃(AD-DROP),它随机丢弃一些高归因位置,以鼓励模型通过更多地依赖低归因位置来减少过度拟合来进行预测。我们还开发了交叉调整策略,以交替微调和AD-DROP,以避免过度丢弃高归因位置。在各种基准测试上的大量实验表明,AD-DROP在基线上产生了一致的改进。分析进一步证实,AD-DROP作为一个战略正则化,以防止过度拟合在微调。

英文摘要:

Fine-tuning large pre-trained language models on downstream tasks is apt to suffer from overfitting when limited training data is available. While dropout proves to be an effective antidote by randomly dropping a proportion of units, existing research has not examined its effect on the self-attention mechanism. In this paper, we investigate this problem through self-attention attribution and find that dropping attention positions with low attribution scores can accelerate training and increase the risk of overfitting. Motivated by this observation, we propose Attribution-Driven Dropout (AD-DROP), which randomly discards some high-attribution positions to encourage the model to make predictions by relying more on low-attribution positions to reduce overfitting. We also develop a cross-tuning strategy to alternate fine-tuning and AD-DROP to avoid dropping high-attribution positions excessively. Extensive experiments on various benchmarks show that AD-DROP yields consistent improvements over baselines. Analysis further confirms that AD-DROP serves as a strategic regularizer to prevent overfitting during fine-tuning.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑