CART: 上下文锚定循环Transformer——一种具有学习稳定性的参数高效架构
CART: Context-Anchored Recurrent Transformer -- A Parameter-Efficient Architecture with Learned Stability
AI总结:
提出CART,一种通过共享核心块循环和冻结键值张量实现参数高效的语言模型,并引入线性时不变门控保持稳定性,实验表明在参数匹配时性能略低于密集基线。
摘要:
我们提出CART(上下文锚定循环Transformer),一种参数高效的语言模型,它在深度上重复使用单个共享核心块R次。与先前每次迭代重新计算键值张量的循环Transformer不同,CART从多层前奏中一次性计算K和V,并通过多头潜在注意力让循环核心交叉关注这些冻结的张量。一个学习得到的线性时不变(LTI)门控保持循环稳定性:其谱半径在所有36个完全训练配置中稳定在窄带内(rho在[0.79, 0.83]之间)。我们在单个消费级GPU上分两个阶段评估CART:首先在3000步进行64配置筛选,然后对36个配置(P=6,R∈{6,8,10},三个种子)训练30500步(约10亿token)。在宽度d∈{256,512,768,1024}上,两个模式成立:前奏深度P主导循环次数R,并且R的第一阶段排名在完全训练时反转(在d≥512时R=6变为最佳)。在绑定d=1024的参数对比测试中,CART未能击败参数匹配的密集基线,在存储参数对比中损失1-2%,在有效参数对比中损失约10%。诊断消融将有效参数差距分为约5%来自权重共享和约5%来自异质的前奏/锚点/核心/尾声框架;循环核心机制(超连接、LTI门控、循环索引嵌入)单独来看是退化的。变R推理在训练R的两侧性能下降,这是该方案下测试时深度扩展的一个负面结果。
英文摘要:
We present CART (Context-Anchored Recurrent Transformer), a parameter-efficient language model that reuses a single shared core block R times across depth. Unlike prior looped transformers that recompute key-value tensors at every iteration, CART computes K and V once from a multi-layer prelude and has the recurrent core cross-attend to those frozen tensors via multi-head latent attention. A learned Linear Time-Invariant (LTI) gate keeps the recurrence stable: its spectral radius settles in a narrow band (rho in [0.79, 0.83]) across all 36 fully-trained configurations. We evaluate CART on single consumer GPUs in two stages: a 64-configuration screen at 3,000 steps, then 36 configurations (P=6, R in {6,8,10}, three seeds) trained for 30,500 steps (~1B tokens). Two patterns hold across widths d in {256,512,768,1024}: prelude depth P dominates loop count R, and the Stage-1 ranking of R reverses at full training (R=6 becomes best at d>=512). At the binding d=1024 parameter-parity test, CART does not beat a parameter-matched dense baseline, losing by 1-2% at stored-parameter parity and by ~10% at effective-parameter parity. Diagnostic ablations split the effective-parameter gap into ~5% from weight sharing and a residual ~5% from the heterogeneous prelude/anchor/core/coda framing; the recurrent-core machinery (hyper-connections, LTI gate, loop-index embedding) is individually vestigial. Variable-R inference degrades on both sides of the trained R, a negative result for test-time depth scaling under this recipe.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。