发表机构
Tsinghua University（清华大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
CyFA是一种线性RNN，通过相对时间分区内存和循环流注意力组织键值关联，在匹配状态大小下提升回忆密集型任务性能并保持高效计算。
AI 中文摘要
线性RNN提供线性时间的序列处理和恒定内存的解码，但其固定大小的循环状态必须容纳所有过去的键值关联。现有的遗忘机制和Delta规则更新通过选择性清除或修正状态来减少干扰，但较早的关联仍可能变得难以检索。我们引入了CyFA（循环流注意力），一种具有相对时间分区内存的线性RNN。在每一步，一个学习的时钟控制着循环传输，该传输在当前键值对进入零年龄槽之前联合应用于键和值状态，从而将存储的关联组织到相对时间槽中。我们进一步推导出绝对时钟坐标的精确变化，将CyFA表示为两个标量衰减线性注意力循环，并实现高效的块状训练。在匹配循环状态大小的400M至1.4B预训练实验中，CyFA提高了回忆密集型性能，同时保持了竞争力的语言建模和高计算效率。在400M规模下，CyFA在FDA上优于KDA（42.60对比26.07），同时仅需KDA前向和后向核心算子执行时间的46.7%和48.3%。我们的代码公开于\href{此URL}{此URL}。
英文摘要
Linear RNNs offer linear-time sequence processing and constant-memory decoding, but their fixed-size recurrent states must accommodate all past key--value associations. Existing forgetting mechanisms and Delta Rule updates reduce interference by selectively clearing or correcting the state, yet earlier associations can still become difficult to retrieve. We introduce CyFA (Cyclic Flow Attention), a Linear RNN with relative-time-partitioned memory. At each step, a learned clock controls the cyclic transport applied jointly to the key and value states before the current key--value pair enters the age-zero slot, thereby organizing stored associations across relative-time slots. We further derive an exact change to absolute-clock coordinates that expresses CyFA as two scalar-decay linear attention recurrences and enables efficient chunk-wise training. Across 400M--1.4B pretraining experiments with matched recurrent-state sizes, CyFA improves recall-intensive performance while maintaining competitive language modeling and high computational efficiency. At 400M, CyFA outperforms KDA on FDA (42.60 vs. 26.07) while requiring only 46.7% and 48.3% of KDA's forward and backward core-operator execution times, respectively. Our code is publicly available at \href{https://github.com/Chyxx/CyclicFlowAttention}{this https URL}.
