arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2610.19625cs.CLcs.AIcs.LG
收藏

DAGFormer:用于Transformer跨层连接的全局拓扑预测器

DAGFormer: Global Topology Prediction for Transformer Cross-Layer Connections

University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校)

Xiaocong Yang, Yuren Hao, ChengXiang Zhai

AI总结:

DAGFormer是一种Transformer语言模型,通过联合训练的拓扑预测器全局规划跨层连接,在多个参数规模、评估任务及结构化剪枝场景下均优于普通Transformer和现有路由基线。

摘要:

我们提出了DAGFormer,这是一种Transformer语言模型,其跨层连接由联合训练的拓扑预测器规划。除了标准Transformer中的残差连接外,DAGFormer还添加了从早期层到后期层的直接连接,预测器会根据输入令牌控制每个连接的强度。与现有分别设置每层连接的跨层路由架构不同,DAGFormer一次性规划整个模型的路由。在参数规模从75M到1B的匹配OLMo-2模型对中,DAGFormer相比普通Transformer将WikiText-2困惑度降低了17%至25%;在5B令牌上训练的1B参数DAGFormer,其性能优于在10B令牌上训练的普通模型。在相同数据和预算下与四个路由基线对比时,DAGFormer在75M、300M和1B参数规模下均达到最低困惑度。该优势还延伸至13项常识、科学、阅读和数学评估任务(在300M和1B参数规模下平均准确率高出2.0至2.5个百分点),以及指令微调任务(相比普通Transformer在AlpacaEval上的胜率为63%至70%,优于所有路由基线);在结构化剪枝下,DAGFormer在6个设置中的5个里性能下降最小。

英文摘要:

We propose DAGFormer, a Transformer language model whose cross-layer connections are planned by a jointly trained topology predictor. Beyond the residual connections in a standard Transformer, DAGFormer adds direct connections from earlier layers to later ones, and the predictor controls the strength of every connection based on input tokens. Unlike existing cross-layer routing architectures, which set each layer's connections separately, DAGFormer plans the routing of the whole model at once. On matched OLMo-2 pairs from 75M to 1B parameters, DAGFormer lowers WikiText-2 perplexity by 17% to 25% over the vanilla Transformer, and a 1B DAGFormer trained on 5B tokens outperforms the vanilla model trained on 10B. Against four routing baselines with the same data and budget, it reaches the lowest perplexity at 75M, 300M and 1B. The gain carries over to 13 commonsense, science, reading and math evaluation tasks (2.0 to 2.5 points higher mean accuracy at 300M and 1B) and to instruction finetuning (63% to 70% AlpacaEval win rate against the vanilla Transformer, leading every routing baseline), and under structured pruning DAGFormer degrades the least in five of six settings.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑