发表机构
School of Future Technology, Xi’an Jiaotong University; National University of Singapore（西安交通大学未来技术学院; 新加坡国立大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
该研究针对AdamW优化器，建立有限时域ISO系统模型，推导符号响应算子与多步误差分解，通过实验验证小批量扰动的延迟影响机制，为优化器动力学分析提供了理论与实证支撑。
AI 中文摘要
由于AdamW在其优化器状态中存储了过去的梯度信息，小批量数据会对训练产生超出其被观测时的更新步骤的影响。我们通过仅在一次梯度更新上存在差异、后续训练序列相同的配对轨迹来研究这种延迟效应。我们将AdamW建模为有限时域输入-状态-输出（ISO）系统，其状态包含模型参数以及一阶和二阶矩估计。对联合动力学进行线性化后得到一个符号响应算子，该算子将局部梯度扰动映射到其未来的损失效应，揭示了优化器记忆如何影响这些效应的幅度、时间和符号。我们进一步推导了精确的多步误差分解，并在局部平滑性和受控激活切换的条件下建立了一阶有限时域精度。实验验证了该响应机制和优化器状态的效应，而重复未来分析则揭示了延迟影响中存在的大量前瞻性结构，这些结构可从ISO近似中部分恢复。代码可在此https URL获取。
英文摘要
A minibatch can influence training beyond the update at which it is observed because AdamW stores past gradient information in its optimizer states. We study this delayed effect through paired trajectories that differ only in one gradient update and share the same subsequent training sequence. We formulate AdamW as a finite-horizon input--state--output (ISO) system whose state contains the model parameters and first- and second-moment estimates. Linearizing the joint dynamics yields a signed response operator that maps a localized gradient perturbation to its future loss effects, revealing how optimizer memory shapes their magnitude, timing, and sign. We further derive an exact multistep error decomposition and establish first-order finite-horizon accuracy under local smoothness and controlled activation switching. Experiments validate the response mechanism and optimizer-state effects, while repeated-future analyses reveal substantial prospective structure in delayed influence that can be partially recovered from ISO approximations. Code is available at https://github.com/Kanyooo/Loss_ISO.
