arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2305.04573cs.CL
收藏

HiFi:高信息注意力头用于参数高效模型适配

HiFi: High-Information Attention Heads Hold for Parameter-Efficient Model Adaptation

Xiamen University(厦门大学)

Anchun Gui, Han Xiao

AI总结:

本文提出参数高效微调方法HiFi,仅微调信息丰富且相关性强的注意力头,通过图建模和PageRank算法识别重要头部,在GLUE基准上取得最优性能。

摘要:

为了在下游任务中充分利用大规模预训练语言模型(PLMs)的优势,对PLMs的全部参数进行微调已成为一种普遍采用的适配范式。然而,由于PLMs参数规模庞大,这种范式在数据稀缺和资源受限的场景下会带来更新效率低下和资源过度消耗的问题。为缓解这些问题,本文提出了一种参数高效的微调方法HiFi,即仅对特定任务中信息量丰富且相关性强的注意力头进行微调。为搜索这些重要的注意力头,我们开发了一个新颖的框架来分析头部的有效性。具体而言,我们首先从信息丰富度和相关性两个角度将头部之间的关系建模为图,然后应用PageRank算法来确定每个头部的相对重要性。在GLUE基准上的大量实验证明了我们方法的有效性,并表明HiFi在先前基线方法上取得了最先进的性能。

英文摘要:

To fully leverage the advantages of large-scale pre-trained language models (PLMs) on downstream tasks, it has become a ubiquitous adaptation paradigm to fine-tune the entire parameters of PLMs. However, this paradigm poses issues of inefficient updating and resource over-consuming for fine-tuning in data-scarce and resource-limited scenarios, because of the large scale of parameters in PLMs. To alleviate these concerns, in this paper, we propose a parameter-efficient fine-tuning method HiFi, that is, only the highly informative and strongly correlated attention heads for the specific task are fine-tuned. To search for those significant attention heads, we develop a novel framework to analyze the effectiveness of heads. Specifically, we first model the relationship between heads into a graph from two perspectives of information richness and correlation, and then apply PageRank algorithm to determine the relative importance of each head. Extensive experiments on the GLUE benchmark demonstrate the effectiveness of our method, and show that HiFi obtains state-of-the-art performance over the prior baselines.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑