arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2304.04947cs.CL
收藏

Conditional Adapters:具有快速推理的参数高效迁移学习

Conditional Adapters: Parameter-efficient Transfer Learning with Fast Inference

Google(谷歌)

Tao Lei, Junwen Bai, Siddhartha Brahma, Joshua Ainslie, Kenton Lee, Yanqi Zhou, Nan Du, Vincent Y. Zhao, Yuexin Wu, Bo Li, Yu Zhang, Ming-Wei Chang

AI总结:

提出Conditional Adapter(CoDA),一种结合条件计算与稀疏激活的参数高效迁移学习方法,在保持参数效率的同时实现2至8倍推理加速且几乎无精度损失。

摘要:

我们提出了Conditional Adapter(CoDA),这是一种参数高效的迁移学习方法,同时提升了推理效率。CoDA推广了标准的Adapter方法,通过条件计算实现了一种平衡速度与准确率的新方式。从一个现有的稠密预训练模型开始,CoDA加入了稀疏激活、少量新参数以及一个轻量级的训练阶段。我们的实验表明,CoDA方法提供了一种出人意料的高效知识迁移方式。在多种语言、视觉和语音任务中,与最先进的Adapter方法相比,CoDA实现了2倍到8倍的推理加速,且准确率损失从适中到没有,并保持了相同的参数效率。

英文摘要:

We propose Conditional Adapter (CoDA), a parameter-efficient transfer learning method that also improves inference efficiency. CoDA generalizes beyond standard adapter approaches to enable a new way of balancing speed and accuracy using conditional computation. Starting with an existing dense pretrained model, CoDA adds sparse activation together with a small number of new parameters and a light-weight training phase. Our experiments demonstrate that the CoDA approach provides an unexpectedly efficient way to transfer knowledge. Across a variety of language, vision, and speech tasks, CoDA achieves a 2x to 8x inference speed-up compared to the state-of-the-art Adapter approaches with moderate to no accuracy loss and the same parameter efficiency.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑