arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2510.04071cs.CL
收藏

是什么让扩散语言模型成为超级数据学习者?

What Makes Diffusion Language Models Super Data Learners?

Ubiquant

Zitian Gao, Haoming Luo, Lynx Chen, Jason Klein Liu, Ran Tao, Joey Zhou, Bryan Dai

AI总结:

本研究通过消融实验揭示扩散语言模型数据效率主要源于输入令牌随机掩码,并发现MLP丢弃和权重衰减等随机正则化也能提升多轮训练的数据效率。

摘要:

近期研究表明,扩散语言模型在有限数据约束下实现了显著的数据效率,但其潜在机制尚不清楚。在本工作中,我们进行了广泛的消融实验,以厘清这种效率的来源。我们的结果表明,输入令牌的随机掩码起着主导作用。我们进一步表明,通过MLP丢弃和权重衰减可以获得类似的收益,这表明随机正则化在多轮训练中广泛增强了数据效率。我们的代码可在https://github.com/zitian-gao/data-efficiency获取。

英文摘要:

Recent studies have shown that diffusion language models achieve remarkable data efficiency under limited-data constraints, yet the underlying mechanisms remain unclear. In this work, we perform extensive ablation experiments to disentangle the sources of this efficiency. Our results show that random masking of input tokens plays the dominant role. We further show that similar gains can be obtained through in MLP dropout and weight decay, indicating that stochastic regularization broadly enhances data efficiency in multi-epoch training. Our code is available at https://github.com/zitian-gao/data-efficiency.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑