用于在线自监督回声状态网络的可扩展扰动学习
Scalable Perturbation Learning for Online Self-Supervised Learning in Echo State Networks
浏览论文内容
中文总结 AI 辅助
研究针对高维系统中智能系统自适应要求间的矛盾，聚焦回声状态网络，提出基于扰动学习成本正交分解的在线自监督学习规则，降低有效扰动维度，避免方差增长，为可扩展及硬件兼容学习提供设计原则。
中文摘要 AI 辅助
智能系统不仅要解决任务，还需在现实世界约束下进行自适应。通过自监督学习实现自主自适应、通过在线学习实现顺序自适应以及通过基于扰动的学习实现内存高效实现，是此类系统的重要要求。然而，对于高维系统，这些要求通常相互矛盾，因为基于扰动的学习存在随扰动变量维度增长的方差问题。本研究聚焦于回声状态网络（ESN），其中在大型储层中自然会出现这种矛盾。我们提出了一种用于ESN在线自监督学习的基于扰动的学习规则。该规则源自自监督学习成本的正交分解，将输入相关分量与由固定ESN参数确定的冗余分量分离。通过仅扰动输入相关分量，有效扰动维度从储层维度降至输入维度。因此，该方法保留了自监督自适应、在线学习和标量反馈扰动学习，同时避免了与储层大小相关的方差增长。这为可扩展且硬件兼容的学习提出了一种设计原则：在线学习应限于目标的动态必要低维分量。
英文摘要
Intelligent systems should not only solve tasks but also adapt under real-world constraints. Autonomous adaptation via self-supervised learning, sequential adaptation via online learning, and memory-efficient implementation via perturbation-based learning are important requirements for such systems. However, these requirements are generally in tension for high-dimensional systems, because perturbation-based learning suffers from variance that grows with the dimension of the perturbed variables. In this study, we focus on echo state networks (ESNs), where this tension naturally arises in large reservoirs. We propose a perturbation-based learning rule for online self-supervised learning in ESNs. The proposed rule is derived from an orthogonal decomposition of the self-supervised learning cost, which separates an input-dependent component from a redundant component determined by the fixed ESN parameters. By perturbing only the input-dependent component, the effective perturbation dimension is reduced from the reservoir dimension to the input dimension. Thus, the proposed method preserves self-supervised adaptation, online learning, and scalar-feedback perturbation learning, while avoiding reservoir-size-dependent variance growth. This suggests a design principle for scalable and hardware-compatible learning: online learning should be restricted to the dynamically necessary low-dimensional component of the objective.
发表机构
- Graduate School of Information Science and Technology, The University of Tokyo（东京大学信息科学与技术研究生院）
机构由 AI 辅助整理，请以论文原文为准。
