arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2406.10418eess.SYcs.SY
收藏

一种用于由线性动态系统生成奖励的上下文随机多臂老虎机自适应方法

An Adaptive Method for Contextual Stochastic Multi-armed Bandits with Rewards Generated by a Linear Dynamical System

Jonathan Gornet, Mehdi Hosseinzadeh, Bruno Sinopoli

AI总结:

本文提出了一种自适应方法,将随机多臂老虎机建模为未知线性动态系统,通过自适应调整模型大小和探索长度来提高累积奖励率。

摘要:

在线决策可以被表述为流行的随机多臂老虎机问题,其中学习者通过做出决策(或采取行动)来最大化从未知环境收集的累积奖励。本文提出将随机多臂老虎机建模为一个未知的线性高斯动态系统,因为许多应用,如动态定价问题中的老虎机或机器学习模型的超参数选择,都可以从中受益。遵循这种方法,我们可以构建一个系统的稳态卡尔曼滤波矩阵表示,该表示利用先前收集的一段时间长度为s的观测数据来预测每个动作将返回的下一个奖励。本文提出了一种解决方案,其中参数s通过一种自适应算法确定,该算法通过分析矩阵表示的模型不确定性来确定。该算法帮助学习者根据其环境模型的不确定性自适应地调整其模型大小和探索长度。通过广泛的数值研究验证了所提出方案的有效性,结果显示所提出的方案能够提高所收集累积奖励的速度。

英文摘要:

Online decision-making can be formulated as the popular stochastic multi-armed bandit problem where a learner makes decisions (or takes actions) to maximize cumulative rewards collected from an unknown environment. This paper proposes to model a stochastic multi-armed bandit as an unknown linear Gaussian dynamical system, as many applications, such as bandits for dynamic pricing problems or hyperparameter selection for machine learning models, can benefit from this perspective. Following this approach, we can build a matrix representation of the system's steady-state Kalman filter that takes a set of previously collected observations from a time interval of length $s$ to predict the next reward that will be returned for each action. This paper proposes a solution in which the parameter $s$ is determined via an adaptive algorithm by analyzing the model uncertainty of the matrix representation. This algorithm helps the learner adaptively adjust its model size and its length of exploration based on the uncertainty of its environmental model. The effectiveness of the proposed scheme is demonstrated through extensive numerical studies, revealing that the proposed scheme is capable of increasing the rate of collected cumulative rewards.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑