arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2605.10643cs.CLcs.LG
收藏

单层模型也能进行语言建模

A Single-Layer Model Can Do Language Modeling

Proton

Zanmin Wang

AI总结:

本文提出单层接地预测网络GPN,通过单个递归模块实现语言建模,实验表明其在参数量为1.3亿时达到FineWeb-Edu的18.06 perplexity,接近深度模型表现。

摘要:

现代语言模型通过堆叠层来扩展深度,每层持有自己的状态——transformers中的每层KV缓存,Mamba中的每层矩阵,Gated DeltaNet (GDN)、RWKV和xLSTM中的每层矩阵。生物系统更依赖递归而非堆叠。我们探讨这种形状在语言建模中的潜力。我们提出Grounded Prediction Networks (GPN):通过单个递归模块重新访问状态向量——一个FFN,一个共享矩阵内存。在1.3亿参数下,1层GPN+M达到FineWeb-Edu的18.06 perplexity,接近12层Transformer++ (16.05)和10层GDN (15.34);2层变体缩小差距至6%/11%。我们不匹配深度基线。因为工作上下文是单个向量,我们可以直接检查其几何:持久的默认标记方向,承载内容的数十个标记视野,以及记忆头会自发分裂成快速和慢速保留池。

英文摘要:

Modern language models scale depth by stacking layers, each holding its own state - a per-layer KV cache in transformers, a per-layer matrix in Mamba, Gated DeltaNet (GDN), RWKV, and xLSTM. Biological systems lean heavily on recurrence rather than on stacking. We ask how far that shape can go on language modeling. We propose Grounded Prediction Networks (GPN): one state vector revisited at every step through a single recurrent block - one FFN, one shared matrix memory. At 130M parameters, a 1-layer GPN+M reaches FineWeb-Edu perplexity 18.06, within 13% of a 12-layer Transformer++ (16.05) and 18% of a 10-layer GDN (15.34); a 2-layer variant closes the gap to 6%/11%. We do not match the deep baselines. Because the working context is a single vector, we can directly inspect its geometry: a persistent default-token direction, a content-bearing horizon of tens of tokens, and memory heads that split spontaneously into fast and slow retention pools.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑