arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2211.01979cs.CLcs.AIcs.LG
收藏

Tiny-Attention适配器:上下文比参数的数量更重要

Tiny-Attention Adapter: Contexts Are More Important Than the Number of Parameters

University of Chicago(芝加哥大学);Adobe Research(奥多比研究中心);TTI-Chicago(芝加哥丰田技术研究所)

Hongyu Zhao, Hao Tan, Hongyuan Mei

摘要:

适配器调优是通过添加和调优少量新参数将预训练的语言模型转移到下游任务的范例。以前提出的适配器架构都是前馈神经网络。在本文中,我们研究了使用微小注意力的有效性,即,注意,每个头的尺寸非常小-作为适配器。我们的tiny-attention适配器学习修改隐藏状态在每个位置直接条件的隐藏状态在所有其他位置,这是错过了以前提出的适配器。此外,我们认为它的多个注意头作为一个混合专家模型,并建议平均他们的权重在部署过程中,这进一步降低了其推理计算成本。在GLUE基准测试中,我们的tiny-attention适配器优于其他参数高效的迁移学习方法以及完全微调,同时只更新0.05%的参数。在FewGLUE基准测试中,其性能与GPT-3和PET相当。

英文摘要:

Adapter-tuning is a paradigm that transfers a pretrained language model to downstream tasks by adding and tuning a small number of new parameters. Previously proposed adapter architectures are all feed-forward neural networks. In this paper, we investigate the effectiveness of using tiny-attention -- i.e., attention with extremely small per-head dimensionality -- as adapters. Our tiny-attention adapter learns to modify the hidden states at each position directly conditioned on the hidden states at all the other positions, which is missed by the previously proposed adapters. Moreover, we view its multiple attention heads as a mixture of experts and propose to average their weights during deployment, which further reduces its inference computation cost. On the GLUE benchmark, our tiny-attention adapter outperforms the other parameter-efficient transfer learning methods as well as full fine-tuning while only updating 0.05% of the parameters. On the FewGLUE benchmark, its performance is comparable to that of GPT-3 and PET.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑