arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2411.01035cs.LGcs.AIcs.CL
收藏

基于谱滤波的序列预测中可证明的长度泛化

Provable Length Generalization in Sequence Prediction via Spectral Filtering

Google DeepMind(谷歌DeepMind)

Annie Marsden, Evan Dogariu, Naman Agarwal, Xinyi Chen, Daniel Suo, Elad Hazan

AI总结:

该研究针对序列预测中的长度泛化问题,定义非对称遗憾度量,提出基于谱滤波的梯度学习算法,证明其在线性动力系统上可实现长度泛化,并通过概念验证实验验证了理论结论。

摘要:

我们研究序列预测中的长度泛化问题。我们定义了该场景下一种新的性能度量——Asymmetric-Regret(非对称遗憾),用于衡量与上下文长度长于学习者可用长度的基准预测器相比的遗憾值。我们接着通过spectral filtering(谱滤波)算法的视角研究这一概念,提出了一种基于梯度的学习算法,该算法可证明能在线性动力系统上实现长度泛化。最后我们开展了与理论相符的概念验证实验。

英文摘要:

We consider the problem of length generalization in sequence prediction. We define a new metric of performance in this setting -- the Asymmetric-Regret -- which measures regret against a benchmark predictor with longer context length than available to the learner. We continue by studying this concept through the lens of the spectral filtering algorithm. We present a gradient-based learning algorithm that provably achieves length generalization for linear dynamical systems. We conclude with proof-of-concept experiments which are consistent with our theory.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑