发表机构
Google DeepMind; YouTube（谷歌DeepMind; 优兔）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
ClockRoPE基于随机傅里叶旋转理论提出，用于序列推荐的时序常规建模，经在线A/B测试可提升参与度指标，已部署于大型视频平台的生成式检索系统。
AI 中文摘要
旋转位置嵌入（RoPE）已被广泛应用于基于Transformer的大语言模型中。然而，其最初为产生长期注意力衰减而设计的对数线性频率调度，限制了它在具有更复杂距离-相关性模式的领域中的应用，比如序列推荐中的时序周期性。我们研究了通用查询/键旋转的表达能力，发现任何归一化的连续正定注意力调制函数都可由其自身傅里叶变换诱导的随机旋转近似，我们将其称为随机傅里叶旋转。基于该理论，我们提出了ClockRoPE用于序列推荐中的常规建模，其旋转频率源自周期性注意力调制函数。在在线A/B测试中，ClockRoPE在重要的参与度指标上展现出持续的提升，且已成功部署在某大型视频分享平台的生产级生成式检索系统中。
英文摘要
Rotary Position Embedding (RoPE) has been widely adopted in transformer-based large language models. However, its log-linear frequency schedule, originally designed to produce long-term attention decay, limits its adoption in domains with more complex distance-correlation patterns, such as temporal periodicity in sequential recommendation. We investigate the expressiveness of general query/key rotations and find that any normalized continuous positive-definite attention modulation function can be approximated by random rotations induced by its own Fourier transform, which we term Random Fourier Rotations. Building on this theory, we propose ClockRoPE for routine modeling in sequential recommendation, where rotation frequencies are derived from periodic attention modulation functions. In online A/B tests, ClockRoPE demonstrates consistent improvements in valued engagement metrics, and has been successfully deployed in production-scale generative retrieval system at a major video-sharing platform.
