图机：通过边实现更好的预训练
Graph Machine: Towards Better Pretraining via Edges
浏览论文内容
中文总结 AI 辅助
提出图机（GM）架构，替换Qwen3-0.6B中75%的密集Transformer层为GM稀疏层，在157亿token上预训练，仅用少量检索token即可小幅改善损失。
中文摘要 AI 辅助
我们提出了图机（Graph Machine, GM），这是一种保持O(n)规模状态并通过稀疏动态路由访问该状态的架构。与具有固定大小状态或稀疏但静态路由的方法不同，GM在其稀疏层中保持O(n)复杂度，同时不将潜在可访问的状态大小限制为O(1)。相反，GM使用类似指针的边对象，该对象通过类似指针追逐的引用机制进行可微分更新。我们将Qwen3-0.6B中的75%的密集Transformer层替换为GM稀疏层，并在157亿个token上从头开始预训练。在每个稀疏层中，每个KV头仅检索4096个token中的2个时，损失仅略有下降；当检索4个时，最佳模型的损失略有改善。
英文摘要
We introduce the Graph Machine (GM), an architecture that maintains an $O(n)$-sized state and accesses it through sparse, dynamic routing. Unlike methods with fixed-size states or sparse but static routing, GM preserves $O(n)$ complexity in its sparse layers without restricting the potentially accessible state size to $O(1)$. Instead, GM uses edges - pointer-like objects updated differentiably by a referral mechanism resembling pointer chasing. We replace 75% of the dense Transformer layers in Qwen3-0.6B with GM sparse layers and pretrain from scratch on 15.7B tokens. With only 2 of 4,096 tokens retrieved per KV head in each sparse layer, loss degrades only slightly; with 4, the best model marginally improves loss.
