arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2305.14760cs.CL
收藏

Bi-Drop:通过同步子网络估计与优化增强微调泛化能力

Bi-Drop: Enhancing Fine-tuning Generalization via Synchronous sub-net Estimation and Optimization

Peking University(北京大学);Alibaba Group(阿里巴巴集团);Tencent Cloud AI(腾讯云人工智能)

Shoujie Tong, Heming Xia, Damai Dai, Runxin Xu, Tianyu Liu, Binghuai Lin, Yunbo Cao, Zhifang Sui

AI总结:

本文提出微调策略Bi-Drop,利用dropout动态生成的子网络梯度选择性地更新参数,通过同步批内估计克服更新滞后并提升数据利用率,在GLUE基准上表现优异且在领域迁移等场景下具有强泛化能力。

摘要:

预训练语言模型在自然语言理解中取得了显著成功。然而,在有限的训练数据上微调预训练模型往往会导致过拟合,从而降低性能。本文提出了Bi-Drop,这是一种微调策略,它利用由dropout动态生成的各种子网络的梯度来选择性地更新模型参数。Bi-Drop的子网络估计以批内方式进行,因此克服了以往执行异步子网络估计的方法在子网络更新中存在的滞后问题。此外,Bi-Drop仅需一个mini-batch即可估计子网络,从而实现了更高的训练数据利用率。在GLUE基准上的实验表明,Bi-Drop持续优于先前的微调方法。进一步地,实证结果还表明,Bi-Drop在领域迁移、数据不平衡和低资源场景下展现出卓越的泛化能力和鲁棒性。

英文摘要:

Pretrained language models have achieved remarkable success in natural language understanding. However, fine-tuning pretrained models on limited training data tends to overfit and thus diminish performance. This paper presents Bi-Drop, a fine-tuning strategy that selectively updates model parameters using gradients from various sub-nets dynamically generated by dropout. The sub-net estimation of Bi-Drop is performed in an in-batch manner, so it overcomes the problem of hysteresis in sub-net updating, which is possessed by previous methods that perform asynchronous sub-net estimation. Also, Bi-Drop needs only one mini-batch to estimate the sub-net so it achieves higher utility of training data. Experiments on the GLUE benchmark demonstrate that Bi-Drop consistently outperforms previous fine-tuning methods. Furthermore, empirical results also show that Bi-Drop exhibits excellent generalization ability and robustness for domain transfer, data imbalance, and low-resource scenarios.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑