Fine-tuning large pre-trained models with task-specific data has achieved great success in NLP. However, it has been demonstrated that the majority of information within the self-attention networks is redundant and not utilized effectively during the fine-tuning stage. This leads to inferior results when generalizing the obtained models to out-of-domain distributions. To this end, we propose a simple yet effective data augmentation technique, HiddenCut, to better regularize the model and encourage it to learn more generalizable features. Specifically, contiguous spans within the hidden space are dynamically and strategically dropped during training. Experiments show that our HiddenCut method outperforms the state-of-the-art augmentation methods on the GLUE benchmark, and consistently exhibits superior generalization performances on out-of-distribution and challenging counterexamples. We have publicly released our code at https://github.com/GT-SALT/HiddenCut.
ChatGPT as Data Augmentation for Compositional Generalization: A Case Study in Open Intent Detection
Ingenuity Labs Research Institute(创新实验室研究所);Queen’s University(女王大学);Smith School of Business(史密斯商学院);Department of Electrical and Computer Engineering(电气与计算机工程系)
Generative Data Augmentation for Commonsense Reasoning
Northwestern University(西北大学);Allen Institute for Artificial Intelligence(艾伦人工智能研究所);University of Pennsylvania(宾夕法尼亚大学);Carnegie Mellon University(卡内基梅隆大学);University of Washington(华盛顿大学)
MuggleMath: Assessing the Impact of Query and Response Augmentation on Math Reasoning
Alibaba Group(阿里巴巴集团);University of Science and Technology of China(中国科学技术大学);Institute of Dataspace, Hefei Comprehensive National Science Center(合肥综合性国家科学中心天枢研究院)