HyPe:通过隐藏表示扰动实现更好的预训练语言模型微调
HyPe: Better Pre-trained Language Model Fine-tuning with Hidden Representation Perturbation
AI总结:
HyPe通过扰动Transformer层隐藏表示来缓解预训练语言模型微调中的过拟合和表示坍缩问题,在GLUE等数据集上优于普通微调,且计算开销极小。
摘要:
基于Transformer结构的语言模型在自然语言处理中表现出色。然而,在下游任务上微调预训练语言模型时仍存在问题,如过拟合或表示坍缩。在这项工作中,我们提出了HyPe,一种简单而有效的微调技术,通过扰动Transformer层的隐藏表示来缓解这些问题。与以往仅向输入或参数添加噪声的工作不同,我们认为Transformer层的隐藏表示传达了更多样化和有意义的语言信息。因此,使Transformer层对隐藏表示扰动更具鲁棒性可以进一步整体上有利于PLMs的微调。我们在GLUE和其他自然语言推理数据集上进行了大量实验和分析。结果表明,HyPe优于普通微调,并增强了不同层隐藏表示的泛化能力。此外,HyPe仅需可忽略的计算开销,并且优于且兼容于先前最先进的微调技术。
英文摘要:
Language models with the Transformers structure have shown great performance in natural language processing. However, there still poses problems when fine-tuning pre-trained language models on downstream tasks, such as over-fitting or representation collapse. In this work, we propose HyPe, a simple yet effective fine-tuning technique to alleviate such problems by perturbing hidden representations of Transformers layers. Unlike previous works that only add noise to inputs or parameters, we argue that the hidden representations of Transformers layers convey more diverse and meaningful language information. Therefore, making the Transformers layers more robust to hidden representation perturbations can further benefit the fine-tuning of PLMs en bloc. We conduct extensive experiments and analyses on GLUE and other natural language inference datasets. Results demonstrate that HyPe outperforms vanilla fine-tuning and enhances generalization of hidden representations from different layers. In addition, HyPe acquires negligible computational overheads, and is better than and compatible with previous state-of-the-art fine-tuning techniques.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。