arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2207.09099cs.CLcs.LG
收藏

语言模型的Bagging方法分析

Analyzing Bagging Methods for Language Models

New York University(纽约大学)

Pranab Islam, Shaan Khosla, Arthur Lok, Mudit Saxena

摘要:

现代语言模型利用越来越多的参数来实现自然语言理解任务的性能。将这些模型集成到下游任务的特定配置中,可以进一步提高性能。在本文中,我们进行了装袋语言模型的分析,并比较单一的语言模型装袋合奏是大致相等的最终模型的大小。我们探索了一系列用于自然语言理解任务的模型装袋配置,最终集成大小从300 M参数到1.5 B参数不等,并确定我们的集成方法充其量大致相当于单个LM基线。我们注意到其他积极的影响,装袋和修剪在特定的情况下,根据我们的实验结果,如方差减少和轻微的性能改善。

英文摘要:

Modern language models leverage increasingly large numbers of parameters to achieve performance on natural language understanding tasks. Ensembling these models in specific configurations for downstream tasks show even further performance improvements. In this paper, we perform an analysis of bagging language models and compare single language models to bagged ensembles that are roughly equivalent in terms of final model size. We explore an array of model bagging configurations for natural language understanding tasks with final ensemble sizes ranging from 300M parameters to 1.5B parameters and determine that our ensembling methods are at best roughly equivalent to single LM baselines. We note other positive effects of bagging and pruning in specific scenarios according to findings in our experiments such as variance reduction and minor performance improvements.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑