arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2610.33360cs.LG
收藏

带有共享键值缓存的LSTM风格循环Transformer

LSTM-Style Looped Transformers with Shared KV Caches

Andreas Kirsch

AI总结:

本文提出带共享KV缓存的LSTM风格循环Transformer,通过一致性损失解决训练缓存不匹配问题,在FineWeb上降低延续损失惩罚,为数据与存储有限场景提供循环计算方案。

摘要:

循环Transformer通过复用权重以更少参数实现深度计算。本文研究一种LSTM风格Transformer,其在循环核心中共享一个键值缓存。在宽度为1024时,它比未绑定参数的对照模型少用40.4%的参数,且前向FLOPs几乎匹配,同时将持久缓存集从15个减少到4个。并行训练从标准注意力传递中提供缓存条目,而延续阶段则读取使用共享注意力递归生成的条目。我们通过在固定训练token预算的最后10%期间使用一致性损失,使两个训练传递的缓存条目对齐,以此解决这种不匹配问题。在FineWeb上的三种模型宽度下,这消除了相对于全缓存评估的81%至89%的延续损失惩罚。该改进在两个训练种子的不相交文档上得到复现。在四种宽度和1.05至4.19B训练token下,权重绑定的损失成本在较低token预算时最小,这表明在数据和模型存储有限时,有机会使用循环计算。

英文摘要:

Recurrent transformers reuse weights to perform deep computation with fewer parameters. We study an LSTM-style transformer that also shares one key--value cache across its recurrent core. At width 1024, it uses 40.4% fewer parameters than an untied control at nearly matched forward FLOPs, and reduces persistent cache sets from fifteen to four. Parallel training supplies cache entries from a standard-attention pass, whereas continuation reads entries produced recursively using shared attention. We address this mismatch by aligning the two training passes' cache entries with a consistency loss during the final 10% of a fixed training-token budget. Across three model widths on FineWeb, this removes 81--89% of the continuation-loss penalty relative to full-cache evaluation. The improvement replicates on disjoint documents for two training seeds. Across four widths and 1.05--4.19B training tokens, the loss cost of weight tying is smallest at lower token budgets, suggesting an opportunity to use recurrent computation when data and model storage are limited.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑