arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2606.23957cs.LGcs.SYeess.SYq-bio.MN
收藏

使用无注意力变换器学习Koopman算子

Learning the Koopman Operator using Attention Free Transformers

Institute for Bioengineering, School of Engineering and Centre for Engineering Biology, University of Edinburgh(爱丁堡大学生物工程研究所、工程学院与工程生物学中心);Electrical and Computer Engineering and Applied Mathematics, University of Washington(华盛顿大学电气与计算机工程与应用数学系);Electrical and Computer Engineering and Computer Science & Engineering, University of Washington(华盛顿大学电气与计算机工程及计算机科学与工程系)

Mohammed Nagdi, Evangelos-Marios Nikolados, Alexey Yermakov, Mars Gao, Nathan Kutz, Filippo Menolascina

AI总结:

针对Koopman自编码器长期预测中流形漂移导致误差累积的问题,提出无注意力潜在记忆块和动态重编码两种方法,在三个基准系统上显著降低长时域误差并保持低推理延迟。

摘要:

使用自编码器学习Koopman算子能够在潜在空间中进行线性预测,但长期展开往往偏离学习到的流形,导致在具有切换、连续谱或强瞬态的系统上出现相位和幅度误差。我们引入了两个互补的组件,使Koopman预测器更加鲁棒。首先,我们添加了一个无注意力潜在记忆块,该块聚合过去潜在变量的短窗口,以在每次Koopman更新之前产生校正后的潜在变量。与多头注意力不同,AFT以线性时间运行,仅增加约30k个参数($3d^2 + T^2$,少于匹配的多头注意力),但能够捕获抑制误差发散所需的局部时间上下文。其次,我们提出了动态重编码:轻量级的在线变点触发检测器,用于检测潜在漂移并将预测投影回自编码器流形。在三个基准系统(Duffing振荡器、Repressilator、IRMA)上,与Koopman自编码器和匹配容量的多头注意力相比,我们的模型持续减少了误差累积。我们还与GRU和Transformer自编码器进行了比较,从初始条件和50步上下文两种方式评估,发现Koopman+AFT(可选重编码)在保持较低推理延迟的同时,实现了显著更低的长期误差。我们报告了长达1000步时域上的改进,以及关于触发策略的消融实验。结果是一个快速、紧凑的预测器,能够在长时间范围内保持在学习的流形上。

英文摘要:

Learning Koopman operators with autoencoders enables linear prediction in a latent space, but long-horizon rollouts often drift off the learned manifold, leading to phase and amplitude errors on systems with switching, continuous spectra, or strong transients. We introduce two complementary components that make Koopman predictors more robust. First, we add an attention-free latent memory (AFT) block that aggregates a short window of past latents to produce a corrected latent before each Koopman update. Unlike multi-head attention, AFT operates in linear time and adds only $\approx$30k parameters ($3d^2 + T^2$, fewer than matched multi-head attention), yet captures the local temporal context needed to suppress error divergence. Second, we propose dynamic re-encoding: lightweight, online change-point triggers (EWMA, CUSUM, and sequential two-sample tests) that detect latent drift and project predictions back onto the autoencoder manifold. Across three benchmark systems -- Duffing oscillator, Repressilator, IRMA -- our model consistently reduces error accumulation compared to a Koopman autoencoder and matched-capacity multi-head attention. We also compare against GRU and Transformer autoencoders, evaluated both from initial conditions and with a 50-step context, and find that Koopman+AFT (with optional re-encoding) attains markedly lower long-horizon error while maintaining lower inference latency. We report improvements over horizons up to 1000 steps, together with ablations over trigger policies. The result is a fast, compact predictor that stays on the learned manifold over long horizons.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑