在波动环境中使用贝叶斯因子惊喜的学习
Learning in Volatile Environments with the Bayes Factor Surprise
摘要：
基于惊喜的学习允许代理快速适应以突然变化为特征的非平稳随机环境。我们表明，确切的贝叶斯推断在层次模型中产生了一个模糊调制的权衡之间的遗忘旧的意见，并将它们与新的。调制取决于概率比，我们称之为“贝叶斯因素惊喜”，它测试了当前信念的先验信念。我们证明，在几个现有的近似算法的贝叶斯因子惊喜调制适应新的观察的速度。我们推导出三种新的基于递归的算法，一个在家庭的粒子滤波器，一个在家庭的变分学习，和其他家庭的消息传递，具有恒定的缩放观察序列长度和特别简单的更新动态指数家庭中的任何分布。实证结果表明，这些基于递归的算法估计参数优于替代近似方法，并达到计算上更昂贵的算法的性能水平。贝叶斯因子惊奇与香农惊奇有联系但又有区别。在两个假设的实验中，我们对生理指标进行了可检验的预测，这些指标将贝叶斯因子惊喜与香农惊喜分离开来。将各种方法铸造为基于递归的学习的理论见解，以及所提出的在线算法，可以应用于动物和人类行为的分析，以及非平稳环境中的强化学习。
英文摘要：
Surprise-based learning allows agents to rapidly adapt to non-stationary stochastic environments characterized by sudden changes. We show that exact Bayesian inference in a hierarchical model gives rise to a surprise-modulated trade-off between forgetting old observations and integrating them with the new ones. The modulation depends on a probability ratio, which we call "Bayes Factor Surprise", that tests the prior belief against the current belief. We demonstrate that in several existing approximate algorithms the Bayes Factor Surprise modulates the rate of adaptation to new observations. We derive three novel surprised-based algorithms, one in the family of particle filters, one in the family of variational learning, and the other in the family of message passing, that have constant scaling in observation sequence length and particularly simple update dynamics for any distribution in the exponential family. Empirical results show that these surprise-based algorithms estimate parameters better than alternative approximate approaches and reach levels of performance comparable to computationally more expensive algorithms. The Bayes Factor Surprise is related to but different from Shannon Surprise. In two hypothetical experiments, we make testable predictions for physiological indicators that dissociate the Bayes Factor Surprise from Shannon Surprise. The theoretical insight of casting various approaches as surprise-based learning, as well as the proposed online algorithms, may be applied to the analysis of animal and human behavior, and to reinforcement learning in non-stationary environments.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
