在下游任务上对大型预训练语言模型进行全参数微调的代价极其高昂。因此，参数高效微调引起了广泛关注，该方法在冻结预训练模型的同时仅优化少量任务特定参数。本工作聚焦于前缀微调，它仅优化插入到 Transformer 层中的连续前缀向量（即伪令牌）。基于不同层的学习到的语法和语义表示差异很大的观察，我们论证自适应前缀比固定前缀能进一步适配各层，使微调更有效且高效。为此，我们提出自适应前缀微调（APT），通过门控机制在细粒度令牌级和粗粒度层级上调整前缀。在 SuperGLUE 和 NER 数据集上的实验证明了 APT 的有效性。此外，将门控作为探针，我们验证了可变前缀的高效性与有效性。
英文摘要：
Fine-tuning large pre-trained language models on various downstream tasks with whole parameters is prohibitively expensive. Hence, Parameter-efficient fine-tuning has attracted attention that only optimizes a few task-specific parameters with the frozen pre-trained model. In this work, we focus on prefix tuning, which only optimizes continuous prefix vectors (i.e. pseudo tokens) inserted into Transformer layers. Based on the observation that the learned syntax and semantics representation varies a lot at different layers, we argue that the adaptive prefix will be further tailored to each layer than the fixed one, enabling the fine-tuning more effective and efficient. Thus, we propose Adaptive Prefix Tuning (APT) to adjust the prefix in terms of both fine-grained token level and coarse-grained layer level with a gate mechanism. Experiments on the SuperGLUE and NER datasets show the effectiveness of APT. In addition, taking the gate as a probing, we validate the efficiency and effectiveness of the variable prefix.
Fine-tuning is the de facto way to leverage large pretrained language models to perform downstream tasks. However, it modifies all the language model parameters and therefore necessitates storing a full copy for each task. In this paper, we propose prefix-tuning, a lightweight alternative to fine-tuning for natural language generation tasks, which keeps language model parameters frozen, but optimizes a small continuous task-specific vector (called the prefix). Prefix-tuning draws inspiration from prompting, allowing subsequent tokens to attend to this prefix as if it were "virtual tokens". We …
Australian AI Institute, University of Technology Sydney（悉尼科技大学澳大利亚人工智能研究所）；eBay Inc.（eBay公司）；Microsoft（微软公司）
Recently, prefix-tuning was proposed to efficiently adapt pre-trained language models to a broad spectrum of natural language classification tasks. It leverages soft prefix as task-specific indicators and language verbalizers as categorical-label mentions to narrow the formulation gap from pre-training language models. However, when the label space increases considerably (i.e., many-class classification), such a tuning technique suffers from a verbalizer ambiguity problem since the many-class labels are represented by semantic-similar verbalizers in short language phrases. To overcome this, in…
University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Amazon Alexa AI（亚马逊Alexa人工智能部门）
Prefix-tuning, or more generally continuous prompt tuning, has become an essential paradigm of parameter-efficient transfer learning. Using a large pre-trained language model (PLM), prefix-tuning can obtain strong performance by training only a small portion of parameters. In this paper, we propose to understand and further develop prefix-tuning through the kernel lens. Specifically, we make an analogy between \textit{prefixes} and \textit{inducing variables} in kernel methods and hypothesize that \textit{prefixes} serving as \textit{inducing variables} would improve their overall mechanism. F…
Long-Chain Reasoning Distillation via Adaptive Prefix Alignment
School of Computer Science and Engineering, Northeastern University, China（东北大学计算机科学与工程学院）；Department of Computer Science and Technology, Institute for AI, Tsinghua University, China（清华大学计算机科学与技术系）；Alibaba Group, China（阿里巴巴集团）
L-TUNING: Synchronized Label Tuning for Prompt and Prefix in LLMs
Stevens Institute of Technology（史蒂文斯理工学院）；Hajee Mohammad Danesh Science & Technology University（哈吉·穆罕默德·达内什科学技术大学）；Noakhali Science & Technology University（诺阿卡利科学技术大学）；Amazon（亚马逊）
Beyond Prefixes: Graph-as-Memory Cross-Attention for Knowledge Graph Completion with Large Language Models
Peking University（北京大学）；Dalian University of Technology（大连理工大学）；King’s College London（伦敦国王学院）；Peng Cheng Laboratory（鹏城实验室）；Shanghai Jiao Tong University（上海交通大学）
Singapore University of Technology and Design（新加坡科技设计大学）
Fine-tuning a pre-trained language model via the contrastive learning framework with a large amount of unlabeled sentences or labeled sentence pairs is a common way to obtain high-quality sentence representations. Although the contrastive learning framework has shown its superiority on sentence representation learning over previous methods, the potential of such a framework is under-explored so far due to the simple method it used to construct positive pairs. Motivated by this, we propose a method that makes hard positives from the original training examples. A pivotal ingredient of our approa…
Conventional fine-tuning encounters increasing difficulties given the size of current Pre-trained Language Models, which makes parameter-efficient tuning become the focal point of frontier research. Previous methods in this field add tunable adapters into MHA or/and FFN of Transformer blocks to enable PLMs achieve transferability. However, as an important part of Transformer architecture, the power of layer normalization for parameter-efficent tuning is ignored. In this paper, we first propose LN-tuning, by tuning the gain and bias term of Layer Normalization module with only 0.03\% parameters…
Beijing University of Posts and Telecommunications（北京邮电大学）；Meituan Group（美团集团）
The most advanced abstractive dialogue summarizers lack generalization ability on new domains and the existing researches for domain adaptation in summarization generally rely on large-scale pre-trainings. To explore the lightweight fine-tuning methods for domain adaptation of dialogue summarization, in this paper, we propose an efficient and generalizable Domain-Oriented Prefix-tuning model, which utilizes a domain word initialized prefix module to alleviate domain entanglement and adopts discrete prompts to guide the model to focus on key contents of dialogues and enhance model generalizatio…
Optimizing and Fine-tuning Large Language Model for Urban Renewal
Tsinghua University（清华大学）；Cross-strait Tsinghua Research Institute（海峡两岸清华研究院）；Biocloo Inc（百奥科洛公司）；China Agricultural University（中国农业大学）；Zhipu AI（智谱AI）；Beijing Rongchuang Geotechnical Engineering Co., Ltd.（北京融创岩土工程有限公司）；The University of Hong Kong（香港大学）
Xiamen University（厦门大学）；Ministry of Culture and Tourism（文化和旅游部）
Simultaneous machine translation, which aims at a real-time translation, is useful in many live scenarios but very challenging due to the trade-off between accuracy and latency. To achieve the balance for both, the model needs to wait for appropriate streaming text (READ policy) and then generates its translation (WRITE policy). However, WRITE policies of previous work either are specific to the method itself due to the end-to-end training or suffer from the input mismatch between training and decoding for the non-end-to-end training. Therefore, it is essential to learn a generic and better WR…
Beijing University of Posts and Telecommunications（北京邮电大学）；University of Michigan（密歇根大学）；Singapore University of Technology and Design（新加坡科技设计大学）；Amazon Science（亚马逊科学）
Fine-tuning is widely used as the default algorithm for transfer learning from pre-trained models. Parameter inefficiency can however arise when, during transfer learning, all the parameters of a large pre-trained model need to be updated for individual downstream tasks. As the number of parameters grows, fine-tuning is prone to overfitting and catastrophic forgetting. In addition, full fine-tuning can become prohibitively expensive when the model is used for many tasks. To mitigate this issue, parameter-efficient transfer learning algorithms, such as adapters and prefix tuning, have been prop…
Simple Yet Effective: Extracting Private Data Across Clients in Federated Fine-Tuning of Large Language Models
Harbin Institute of Technology（哈尔滨工业大学）；Kuaishou Technology（快手科技）；Meta AI；Monash University（墨尔本大学）；Fudan University（复旦大学）；Shanghai Academy of Artificial Intelligence for Science（上海人工智能科学研究院）
Large-scale pre-trained language models have achieved impressive results on a wide range of downstream tasks recently. However, fine-tuning an extremely large-scale pre-trained language model on limited target datasets is often plagued by overfitting and representation degradation. In this paper, we propose a Dynamic Parameter Selection (DPS) algorithm for the large-scale pre-trained models during fine-tuning, which adaptively selects a more promising subnetwork to perform staging updates based on gradients of back-propagation. Experiments on the GLUE benchmark show that DPS outperforms previo…
The invention of transformer-based models such as BERT, GPT, and RoBERTa has enabled researchers and financial companies to finetune these powerful models and use them in different downstream tasks to achieve state-of-the-art performance. Recently, a lightweight alternative (approximately 0.1% - 3% of the original model parameters) to fine-tuning, known as prefix tuning has been introduced. This method freezes the model parameters and only updates the prefix to achieve performance comparable to full fine-tuning. Prefix tuning enables researchers and financial practitioners to achieve similar r…
Magnitude pruning is a widely used strategy for reducing model size in pure supervised learning; however, it is less effective in the transfer learning regime that has become standard for state-of-the-art natural language processing applications. We propose the use of movement pruning, a simple, deterministic first-order weight pruning method that is more adaptive to pretrained model fine-tuning. We give mathematical foundations to the method and compare it to existing zeroth- and first-order pruning methods. Experiments show that when pruning large pretrained language models, movement pruning…
Institute for AI Industry Research（人工智能产业研究院）；Tsinghua University（清华大学）
Recently, prefix-tuning has gained increasing attention as a parameter-efficient finetuning method for large-scale pretrained language models. The method keeps the pretrained models fixed and only updates the prefix token parameters for each downstream task. Despite being lightweight and modular, prefix-tuning still lacks robustness to textual adversarial attacks. However, most currently developed defense techniques necessitate auxiliary model update and storage, which inevitably hamper the modularity and low storage of prefix-tuning. In this work, we propose a robust prefix-tuning framework t…
LookAhead Tuning: Safer Language Models via Partial Answer Previews
Zhejiang University（浙江大学）；Zhejiang University - Ant Group Joint Laboratory of Knowledge Graph（浙江大学-蚂蚁集团知识图谱联合实验室）；Ant Group（蚂蚁集团）；National University of Singapore（新加坡国立大学）
Beijing Institute of Technology（北京理工大学）；Beijing University of Technology（北京工业大学）
We consider event extraction in a generative manner with template-based conditional generation. Although there is a rising trend of casting the task of event extraction as a sequence generation problem with prompts, these generation-based methods have two significant challenges, including using suboptimal prompts and static event type information. In this paper, we propose a generative template-based event extraction method with dynamic prefix (GTEE-DynPref) by integrating context information with type-specific prefixes to learn a context-specific prefix for each context. Experimental results …
Dual-Gated Fusion with Prefix-Tuning for Multi-Modal Relation Extraction
Beihang University（北京航空航天大学）；Beijing Advanced Innovation Center for Big Data and Brain Computing（北京大数据与脑计算高精尖创新中心）；The University of Sheffield（谢菲尔德大学）；Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）
Prefix-tuning is a powerful lightweight technique for adapting a large pre-trained language model to a downstream application. However, it uses the same dataset-level tuned prompt for all examples in the dataset. We extend this idea and propose a dynamic method, Control Prefixes, which allows for the inclusion of conditional input-dependent information, combining the benefits of prompt tuning and controlled generation. The method incorporates attribute-level learnable representations into different layers of a pre-trained transformer, allowing for the generated text to be guided in a particula…
DEGAP: Dual Event-Guided Adaptive Prefixes for Templated-Based Event Argument Extraction with Slot Querying
School of Information Management, Jiangxi University of Finance and Economics（江西财经大学信息管理学院）；University of Montreal (UdeM), Department of Computer Science and Operations Research (DIRO)（蒙特利尔大学（UdeM）计算机科学与运筹学系）；School of Computer Science and Engineering, University of Electronic Science and Technology of China（电子科技大学计算机科学与工程学院）；Department of Statistics, Division of the Physical Sciences, the University of Chicago（芝加哥大学统计学系）
