arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2106.00149cs.CL
收藏

HiddenCut:简单的数据增强,用于更好地概括自然语言理解

HiddenCut: Simple Data Augmentation for Natural Language Understanding with Better Generalization

Georgia Institute of Technology(佐治亚理工学院);Microsoft Dynamics 365 AI(微软Dynamics 365人工智能部门)

Jiaao Chen, Dinghan Shen, Weizhu Chen, Diyi Yang

摘要:

使用特定任务数据微调大型预训练模型在NLP中取得了巨大成功。然而,已经证明,自我注意网络中的大部分信息是冗余的,在微调阶段没有得到有效利用。这导致劣质的结果时,推广所获得的模型域外分布。为此,我们提出了一个简单而有效的数据增强技术,HiddenCut,以更好地规范模型,并鼓励它学习更多的泛化功能。具体地,隐藏空间内的连续跨度在训练期间被动态地和策略性地丢弃。实验表明,我们的HiddenCut方法在GLUE基准测试中的性能优于最先进的增强方法,并且在分布外和具有挑战性的反例上始终表现出优越的泛化性能。我们已在https://github.com/GT-SALT/HiddenCut上公开发布了我们的代码。

英文摘要:

Fine-tuning large pre-trained models with task-specific data has achieved great success in NLP. However, it has been demonstrated that the majority of information within the self-attention networks is redundant and not utilized effectively during the fine-tuning stage. This leads to inferior results when generalizing the obtained models to out-of-domain distributions. To this end, we propose a simple yet effective data augmentation technique, HiddenCut, to better regularize the model and encourage it to learn more generalizable features. Specifically, contiguous spans within the hidden space are dynamically and strategically dropped during training. Experiments show that our HiddenCut method outperforms the state-of-the-art augmentation methods on the GLUE benchmark, and consistently exhibits superior generalization performances on out-of-distribution and challenging counterexamples. We have publicly released our code at https://github.com/GT-SALT/HiddenCut.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑