arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2204.04596cs.CL
收藏

通过操纵预训练语言模型的隐藏状态进行分类任务的参数有效调整

Parameter-Efficient Tuning by Manipulating Hidden States of Pretrained Language Models For Classification Tasks

The Chinese University of Hong Kong(香港中文大学);Tencent AI Lab(腾讯人工智能实验室)

Haoran Yang, Piji Li, Wai Lam

摘要:

参数高效调优旨在通过优化一些引入的参数,同时冻结预训练的语言模型(PLM),为下游任务提取知识。连续提示微调就是其中之一,它通过在输入的嵌入中添加少量可训练向量,以其有效性和高效性而备受关注。这一系列方法可以被说明为对PLM内部的隐藏状态进行非线性变换。然而,一个自然的问题被忽略了:隐藏状态可以直接用于分类而不改变它们吗?在本文中,我们的目的是回答这个问题,提出了一个简单的调整方法,只引入了三个可训练的向量。首先,我们整合所有层隐藏状态使用引入的向量。然后,我们将集成的隐藏状态输入到特定于任务的线性分类器来预测类别。这个方案类似于ELMo利用隐藏状态的方式,除了它们将隐藏状态馈送到基于LSTM的模型。虽然我们提出的调整方案是简单的,它实现了类似的性能与提示微调方法,如P-调整和P-调整v2,验证原始隐藏状态包含有用的信息分类任务。此外,我们的方法在时间和参数数量方面优于提示微调。

英文摘要:

Parameter-efficient tuning aims to distill knowledge for downstream tasks by optimizing a few introduced parameters while freezing the pretrained language models (PLMs). Continuous prompt tuning which prepends a few trainable vectors to the embeddings of input is one of these methods and has drawn much attention due to its effectiveness and efficiency. This family of methods can be illustrated as exerting nonlinear transformations of hidden states inside PLMs. However, a natural question is ignored: can the hidden states be directly used for classification without changing them? In this paper, we aim to answer this question by proposing a simple tuning method which only introduces three trainable vectors. Firstly, we integrate all layers hidden states using the introduced vectors. And then, we input the integrated hidden state(s) to a task-specific linear classifier to predict categories. This scheme is similar to the way ELMo utilises hidden states except that they feed the hidden states to LSTM-based models. Although our proposed tuning scheme is simple, it achieves comparable performance with prompt tuning methods like P-tuning and P-tuning v2, verifying that original hidden states do contain useful information for classification tasks. Moreover, our method has an advantage over prompt tuning in terms of time and the number of parameters.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑