arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2409.05423cs.CL
收藏

STLM工程报告:Dropout

STLM Engineering Report: Dropout

A*STAR(新加坡科技研究局);Singapore Management University(新加坡管理大学);Institute of High Performance Computing (IHPC)(高性能计算研究所);Institute of Infocomm Research (I2R)(信息通信研究院)

Dylan Hillier, Leon Guertler, Bobby Cheng, Cheston Tan

AI总结:

本文研究小于100M参数语言模型中的dropout作用,发现其既能缓解小数据过拟合,也可能改善大数据欠拟合,但现有机制解释并不适用于语言建模。

摘要:

在本工作中,我们探讨了dropout对现代语言模型的相关性,特别是在参数规模小于100M的模型背景下。我们首先考察其在给定小型高质量数据集时提升模型样本效率这一情境中的相关性,其次考察其在模型可能欠拟合的较大数据集上提升拟合质量这一情境中的相关性。我们发现,与传统经验一致,dropout在过拟合场景中仍然有效;此外,正如先前研究所表明的,即使在数据充足的情况下,它也可能与提升模型拟合质量存在一定相关性。在此过程中,我们发现,关于这种性能提升背后机制的现有解释并不适用于语言建模场景。

英文摘要:

In this work we explore the relevance of dropout for modern language models, particularly in the context of models on the scale of <100M parameters. We explore it's relevance firstly in the regime of improving the sample efficiency of models given small, high quality datasets, and secondly in the regime of improving the quality of its fit on larger datasets where models may underfit. We find that concordant with conventional wisdom, dropout remains effective in the overfitting scenario, and that furthermore it may have some relevance for improving the fit of models even in the case of excess data, as suggested by previous research. In the process we find that the existing explanation for the mechanism behind this performance gain is not applicable in the case of language modelling.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑