HiPPO-Prophecy：状态空间模型可以在上下文中可证明地学习动态系统
HiPPO-Prophecy: State-Space Models can Provably Learn Dynamical Systems in Context
AI总结：
本文首次从理论上解释了状态空间模型（SSMs）的上下文学习机制，通过扩展HiPPO框架构造显式权重，使SSMs无需微调即可预测任意动态系统的下一状态，并提供了渐近误差界与实证验证。
摘要：
本文探讨了状态空间模型（SSMs）的上下文学习能力，并据我们所知，首次对其可能的底层机制给出了理论解释。我们为SSMs引入了一种新颖的权重构造，使其能够在观察先前状态后预测任何动态系统的下一状态，而无需参数微调。这是通过扩展HiPPO框架来实现的，证明了连续SSMs可以近似任何输入信号的导数。具体而言，我们找到了连续SSMs的显式权重构造，并提供了导数近似的渐近误差界。该连续SSM的离散化随后产生了一个预测下一状态的离散SSM。最后，我们从经验上证明了我们参数化的有效性。这项工作应该是理解基于SSMs的序列模型如何在上下文中学习的初始步骤。
英文摘要：
This work explores the in-context learning capabilities of State Space Models (SSMs) and presents, to the best of our knowledge, the first theoretical explanation of a possible underlying mechanism. We introduce a novel weight construction for SSMs, enabling them to predict the next state of any dynamical system after observing previous states without parameter fine-tuning. This is accomplished by extending the HiPPO framework to demonstrate that continuous SSMs can approximate the derivative of any input signal. Specifically, we find an explicit weight construction for continuous SSMs and provide an asymptotic error bound on the derivative approximation. The discretization of this continuous SSM subsequently yields a discrete SSM that predicts the next state. Finally, we demonstrate the effectiveness of our parameterization empirically. This work should be an initial step toward understanding how sequence models based on SSMs learn in context.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
