一个GPU的数据效率:小型语言模型的转换方法的探索
Data-Efficiency with a Single GPU: An Exploration of Transfer Methods for Small Language Models
摘要:
多任务学习(MTL)、指令微调和提示最近被证明可以提高大语言模型对新任务的泛化能力。然而,这些方法的好处在较小的语言模型中没有得到很好的证明,一些研究发现了相互矛盾的结果。在这项工作中,我们探索和隔离的影响(i)模型大小,(ii)通用MTL,(iii)在域MTL,(iv)指令微调,和(v)Few-Shot微调小于5亿参数的模型。我们在zero-shot设置中的实验表明,模型平均从通用MTL获得31%的相对改进,从域内MTL获得额外的37.6%的相对增益。矛盾的大型模型上的先前的作品,我们发现,指令微调提供了一个温和的2%的性能改进的小型模型。
英文摘要:
Multi-task learning (MTL), instruction tuning, and prompting have recently been shown to improve the generalizability of large language models to new tasks. However, the benefits of such methods are less well-documented in smaller language models, with some studies finding contradictory results. In this work, we explore and isolate the effects of (i) model size, (ii) general purpose MTL, (iii) in-domain MTL, (iv) instruction tuning, and (v) few-shot fine-tuning for models with fewer than 500 million parameters. Our experiments in the zero-shot setting demonstrate that models gain 31% relative improvement, on average, from general purpose MTL, with an additional 37.6% relative gain from in-domain MTL. Contradictory to prior works on large models, we find that instruction tuning provides a modest 2% performance improvement for small models.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。