arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2604.25409cs.CL
收藏

通过高效的跨尺度超参数转移扩展概率变换器

Scaling Probabilistic Transformer via Efficient Cross-Scale Hyperparameter Transfer

School of Information Science and Technology, ShanghaiTech University(信息科学与技术学院,上海科技大学)

Penghao Kuang, Haoyi Wu, Kewei Tu

AI总结:

本文通过maximal update parametrization方法,将概率变换器参数扩展至0.4B规模,在相同参数预算下,MLM任务中性能优于标准变换器。

摘要:

概率变换器(PT),一种用于上下文词表示的白盒概率模型,在计算结构和下游任务性能上与标准变换器在小模型和中小数据集上表现出显著相似性。然而,PT对超参数选择的鲁棒性较差,难以高效扩展。本文遵循maximal update parametrization(muP)方法重新缩放PT的参数,使得在小模型上优化的超参数可以转移到更大的模型而无需额外调优。通过这种方法,我们成功将PT扩展到具有高达0.4B参数的模型。实验表明,在相同参数预算下,PT在掩码语言建模(MLM)任务中 consistently 超过标准变换器。我们希望这项工作将有助于未来在更大规模上部署概率模型。

英文摘要:

Probabilistic Transformer (PT), a white-box probabilistic model for contextual word representation, has demonstrated substantial similarity to standard Transformers in both computational structure and downstream task performance on small models and small to medium sized datasets. However, PT is less robust to hyperparameter choices than standard Transformers, making it harder to scale efficiently. In this work, we follow Maximal Update Parametrization (muP) to rescale PT's parameters, so that hyperparameters optimized on small models can be transferred to larger models without additional tuning. With this approach, we successfully scale PT to models with up to 0.4B parameters. Experiments show that PT consistently outperforms standard transformer under the same parameter budget on Masked Language Modeling (MLM) tasks. We hope this work will contribute to the practical deployment of probabilistic models at substantially larger scales in the future.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑