Recent work has demonstrated that using parameter efficient tuning techniques such as prefix tuning (or P-tuning) on pretrained language models can yield performance that is comparable or superior to fine-tuning while dramatically reducing trainable parameters. Nevertheless, the effectiveness of such methods under the context of data augmentation, a common strategy to improve learning under low data regimes, has not been fully explored. In this paper, we examine the effectiveness of several popular task-agnostic data augmentation techniques, i.e., EDA, Back Translation, and Mixup, when using two general parameter efficient tuning methods, P-tuning v2 and LoRA, under data scarcity. We show that data augmentation can be used to boost the performance of P-tuning and LoRA models, but the effectiveness of each technique varies and certain methods can lead to a notable degradation in performance, particularly when using larger models and on harder tasks. We further analyze the sentence representations of P-tuning compared to fine-tuning to help understand the above behaviour, and reveal how P-tuning generally presents a more limited ability to separate the sentence embeddings from different classes of augmented data. In addition, it displays poorer performance on heavily altered data. However, we demonstrate that by adding a simple contrastive loss function it can help mitigate such issues for prefix tuning, resulting in sizable improvements to augmented data performance.
Better Robustness by More Coverage: Adversarial Training with Mixup Augmentation for Robust Fine-tuning
University of Maryland, College Park（马里兰大学帕克分校）；Tsinghua University（清华大学）；Beijing National Research Center for Information Science and Technology（北京信息科学与技术国家研究中心）；Huawei Noah’s Ark Lab（华为诺亚方舟实验室）
Enhancing LLMs for Identifying and Prioritizing Important Medical Jargons from Electronic Health Record Notes Utilizing Data Augmentation: A Comparative Study
Miner School of Computer and Information Sciences, UMass Lowell, MA, USA（UMass Lowell 计算机与信息科学学院）；Manning College of Information and Computer Sciences, UMass Amherst, MA, USA（UMass Amherst 信息与计算机科学学院）；Center for Healthcare Organization and Implementation Research, VA Bedford Health Care, MA, USA（VA Bedford医疗中心医疗组织与实施研究中心）
LLMs vs Established Text Augmentation Techniques for Classification: When do the Benefits Outweight the Costs?
Faculty of Information Technology, Brno University of Technology（布尔诺理工大学信息技术学院）；Kempelen Institute of Intelligent Technologies（肯佩伦智能技术研究院）；University of Pittsburgh（匹兹堡大学）
A Survey on Data Synthesis and Augmentation for Large Language Models
Hangzhou Innovation Institute, Beihang University（北京航空航天大学杭州创新研究院）；State Key Laboratory of Virtual Reality Technology and Systems, Beihang University（北京航空航天大学虚拟现实技术与系统国家重点实验室）
Minimal-Edit Instruction Tuning for Low-Resource Indic GEC
Indian Heritage Language Computing Team Special and Strategic Projects (SSP) Group（印度遗产语言计算团队特殊与战略项目组）；Centre for Development of Advanced Computing (C DAC)（高级计算发展中心）
On the Effectiveness of Adapter-based Tuning for Pretrained Language Model Adaptation
National University of Singapore（新加坡国立大学）；DAMO Academy, Alibaba Group（阿里巴巴集团达摩院）；Nanyang Technological University（南洋理工大学）；Singapore University of Technology and Design（新加坡科技设计大学）
Span-level Emotion-Cause-Category Triplet Extraction with Instruction Tuning LLMs and Data Augmentation
College of Computer Science and Engineering（计算机科学与工程学院）；Shandong University of Science and Technology（山东科技大学）；Guangxi Key Lab of Human-machine Interaction and Intelligent Decision（广西人机交互与智能决策重点实验室）；Nanning Normal University（南宁师范大学）
Enhancing Effectiveness and Robustness in a Low-Resource Regime via Decision-Boundary-aware Data Augmentation
Chung-Ang University（中央大学）；Graduate School of Advanced Imaging Sciences, Multimedia and Film, Chung-Ang University（中央大学高级影像科学、多媒体与电影研究生院）；Department of Artificial Intelligence, Chung-Ang University（中央大学人工智能系）
MuggleMath: Assessing the Impact of Query and Response Augmentation on Math Reasoning
Alibaba Group（阿里巴巴集团）；University of Science and Technology of China（中国科学技术大学）；Institute of Dataspace, Hefei Comprehensive National Science Center（合肥综合性国家科学中心天枢研究院）
UU-Tax at SemEval-2022 Task 3: Improving the generalizability of language models for taxonomy classification through data augmentation
Utrecht University（乌得勒支大学）；Arab Academy for Science, Technology, and Maritime Transport（阿拉伯科技与海运学院）；Leiden University Medical Center（莱顿大学医学中心）；Leiden University（莱顿大学）
myMediWhisper: Construction of Burmese Medical Speech Corpus and Whisper Fine-Tuning for Clinical Dialogue ASR
National Electronics and Computer Technology Center (NECTEC)（国家电子与计算机技术中心）；King Mongkut’s University of Technology Thonburi（国王蒙固科技大学吞武里分校）；King Mongkut’s Institute of Technology Ladkrabang（国王蒙固科技学院叻甲邦分校）
