Conditional Adapters:具有快速推理的参数高效迁移学习
Conditional Adapters: Parameter-efficient Transfer Learning with Fast Inference
AI总结:
提出Conditional Adapter(CoDA),一种结合条件计算与稀疏激活的参数高效迁移学习方法,在保持参数效率的同时实现2至8倍推理加速且几乎无精度损失。
摘要:
我们提出了Conditional Adapter(CoDA),这是一种参数高效的迁移学习方法,同时提升了推理效率。CoDA推广了标准的Adapter方法,通过条件计算实现了一种平衡速度与准确率的新方式。从一个现有的稠密预训练模型开始,CoDA加入了稀疏激活、少量新参数以及一个轻量级的训练阶段。我们的实验表明,CoDA方法提供了一种出人意料的高效知识迁移方式。在多种语言、视觉和语音任务中,与最先进的Adapter方法相比,CoDA实现了2倍到8倍的推理加速,且准确率损失从适中到没有,并保持了相同的参数效率。
英文摘要:
We propose Conditional Adapter (CoDA), a parameter-efficient transfer learning method that also improves inference efficiency. CoDA generalizes beyond standard adapter approaches to enable a new way of balancing speed and accuracy using conditional computation. Starting with an existing dense pretrained model, CoDA adds sparse activation together with a small number of new parameters and a light-weight training phase. Our experiments demonstrate that the CoDA approach provides an unexpectedly efficient way to transfer knowledge. Across a variety of language, vision, and speech tasks, CoDA achieves a 2x to 8x inference speed-up compared to the state-of-the-art Adapter approaches with moderate to no accuracy loss and the same parameter efficiency.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。