arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2305.07491cs.CL
收藏

适配器效率的全面分析

A Comprehensive Analysis of Adapter Efficiency

Indian Institute of Technology, Madras(马德拉斯印度理工学院);AI4Bharat;National Institute of Information and Communications Technology(国立信息与通信技术研究所);Microsoft(微软)

Nandini Mundra, Sumanth Doddapaneni, Raj Dabre, Anoop Kunchukuttan, Ratish Puduppully, Mitesh M. Khapra

AI总结:

本文通过大量实验证明,适配器在NLU任务中的参数效率并未转化为训练/部署效率优势,建议中等规模模型采用全量微调或多任务训练。

摘要:

适配器被定位为一种参数高效微调(PEFT)方法,即向模型添加最少量的参数并进行微调。然而,适配器尚未得到充分分析,以理解PEFT是否转化为训练/部署效率以及可维护性/可扩展性方面的优势。通过在监督和跨语言零样本设置中对多种适配器、任务和语言进行大量实验,我们清楚地表明,对于自然语言理解(NLU)任务,适配器的参数效率并未转化为相对于模型全量微调的效率提升。更确切地说,适配器训练相对昂贵,且部署延迟略高。此外,适配器的可维护性/可扩展性优势可以通过更简单的方法实现,如通过全量微调进行多任务训练,这些方法还提供相对更快的训练时间。因此,我们建议,对于中等规模的NLU任务模型,从业者应依赖全量微调或多任务训练,而非使用适配器。我们的代码可在https://github.com/AI4Bharat/adapter-efficiency获取。

英文摘要:

Adapters have been positioned as a parameter-efficient fine-tuning (PEFT) approach, whereby a minimal number of parameters are added to the model and fine-tuned. However, adapters have not been sufficiently analyzed to understand if PEFT translates to benefits in training/deployment efficiency and maintainability/extensibility. Through extensive experiments on many adapters, tasks, and languages in supervised and cross-lingual zero-shot settings, we clearly show that for Natural Language Understanding (NLU) tasks, the parameter efficiency in adapters does not translate to efficiency gains compared to full fine-tuning of models. More precisely, adapters are relatively expensive to train and have slightly higher deployment latency. Furthermore, the maintainability/extensibility benefits of adapters can be achieved with simpler approaches like multi-task training via full fine-tuning, which also provide relatively faster training times. We, therefore, recommend that for moderately sized models for NLU tasks, practitioners should rely on full fine-tuning or multi-task training rather than using adapters. Our code is available at https://github.com/AI4Bharat/adapter-efficiency.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑