arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2510.08404cs.CLcs.AIcs.LG
收藏

单层微型Co$^4$超越GPT-2和GPT-BERT

Single layer tiny Co$^4$ outpaces GPT-2 and GPT-BERT

CMI-Lab University of Stirling(CMI实验室斯特林大学)

Noor Ul Zain, Mohsin Raza, Ahsan Adeel

AI总结:

本文提出单层8M参数的Co$^4$模型,以$O(N)$复杂度在10M令牌上仅训练两周期,即在SuperGLUE多数指标上超越更大规模的GPT-2和GPT-BERT,挑战现有深度学习范式与扩展定律。

摘要:

我们展示了一个微型Co$^4$机器(Adeel,2025),其具有单层、两个注意力头和8M参数,以近似$O(N)$的计算成本运行(其中$N$为输入令牌数),仅用两个训练周期就超越了BabyLM挑战基线GPT-2(124M参数,12层,$O(N^2)$)和GPT-BERT(30M参数,12层,$O(N^2)$),而这两个基线均训练了十个周期。Co$^4$在10M令牌上实现了数量级更高的训练效率,展示了高度样本高效的预训练。使用BabyLM挑战评估流程,在复杂基准上,Co$^4$在SuperGLUE任务上展现出强大的零样本和微调性能。具体而言,Co$^4$在7项零样本指标中的5项和7项微调任务中的6项上优于GPT-2,并在两种情况下均于7项指标中的4项上优于GPT-BERT。这些结果表明,有必要重新思考当前主流的深度学习范式及其相关的扩展定律。

英文摘要:

We show that a tiny Co$^4$ machine(Adeel,2025) with a single layer, two heads, and 8M parameters, operating at an approximate cost of $O(N)$ (where $N$ is the number of input tokens), outpaces the BabyLM Challenge baselines GPT-2 (124M, 12 layers, $O(N^2))$ and GPT-BERT (30M, 12 layers, $O(N^2))$ in just two epochs, while both are trained for ten. Co$^4$ achieves orders-of-magnitude greater training efficiency on 10M tokens, demonstrating highly sample efficient pretraining. Using the BabyLM challenge evaluation pipeline across complex benchmarks, Co$^4$ exhibits strong zero-shot and fine-tuning performance on SuperGLUE tasks. Specifically, Co$^4$ outperforms GPT-2 on 5 out of 7 zero-shot metrics and 6 out of 7 fine-tuning tasks, and GPT-BERT on 4 out of 7 metrics in both cases. These results suggest the need to rethink prevailing deep learning paradigms and associated scaling laws.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑