In this paper, we study a non-stationary stochastic bandit problem, which generalizes the switching bandit problem. On top of the switching bandit problem (\textbf{Case a}), we are interested in three concrete examples: (\textbf{b}) the means of the arms are local polynomials, (\textbf{c}) the means of the arms are locally smooth, and (\textbf{d}) the gaps of the arms have a bounded number of inflexion points and where the highest arm mean cannot vary too much in a short range. These three settings are very different, but have in common the following: (i) the number of similarly-sized level sets of the logarithm of the gaps can be controlled, and (ii) the highest mean has a limited number of abrupt changes, and otherwise has limited variations. We propose a single algorithm in this general setting, that in particular solves in an efficient and unified way the four problems (a)-(d) mentioned.
We propose an algorithm for non-stationary kernel bandits that does not require prior knowledge of the degree of non-stationarity. The algorithm follows randomized strategies obtained by solving optimization problems that balance exploration and exploitation. It adapts to non-stationarity by restarting when a change in the reward function is detected. Our algorithm enjoys a tighter dynamic regret bound than previous work on the non-stationary kernel bandit setting. Moreover, when applied to the non-stationary linear bandit setting by using a linear kernel, our algorithm is nearly minimax optim…
A Near-Optimal Change-Detection Based Algorithm for Piecewise-Stationary Combinatorial Semi-Bandits
UIUC（伊利诺伊大学厄巴纳-香槟分校）；SMU（新加坡管理大学）
研究了分段平稳组合半强盗问题。与原来的组合半强盗问题相比，我们的设置假设的奖励分布的基础武器可能会改变在一个分段固定的方式在未知的时间步长。我们提出了一种算法，\texttt{GLR-CUCB}，它采用了一种有效的组合半强盗算法，\texttt{CUCB}，几乎无参数的变点检测器，\emph{Generalized Likelihood Ratio Test}（GLRT）。我们的分析表明，\texttt{GLR-CUCB}的遗憾是上界由$\mathcal{O}(\sqrt{NKT\log{T}})$，其中$N$是分段固定段的数量，$K$是基本臂的数量，和$T$是时间步的数量。作为补充，我们还得到了一个几乎匹配的遗憾下界的顺序$Ω(\sqrt{NKT}$），分段固定的多臂土匪和组合半土匪，使用信息理论技术和明智地构建分段固定的土匪实例。我们的下限比最好的遗憾下限$Ω(\sqrt{T})$更紧。在合成数据集和真实数据集上的数值实验证明了\texttt{GLR-CUCB}算法相对于其他最先进算法的优越性。
Paderborn University（帕德博恩大学）；University of Munich（慕尼黑大学）
We study the non-stationary dueling bandits problem with $K$ arms, where the time horizon $T$ consists of $M$ stationary segments, each of which is associated with its own preference matrix. The learner repeatedly selects a pair of arms and observes a binary preference between them as feedback. To minimize the accumulated regret, the learner needs to pick the Condorcet winner of each stationary segment as often as possible, despite preference matrices and segment lengths being unknown. We propose the $\mathrm{Beat\, the\, Winner\, Reset}$ algorithm and prove a bound on its expected binary weak…
We consider minimisation of dynamic regret in non-stationary bandits with a slowly varying property. Namely, we assume that arms' rewards are stochastic and independent over time, but that the absolute difference between the expected rewards of any arm at any two consecutive time-steps is at most a drift limit $δ> 0$. For this setting that has not received enough attention in the past, we give a new algorithm which extends naturally the well-known Successive Elimination algorithm to the non-stationary bandit setting. We establish the first instance-dependent regret upper bound for slowly varyi…
Naveen Jindal School of Management, University of Texas at Dallas（德克萨斯大学达拉斯分校纳文·金达尔管理学院）
In this paper we study the non-stationary stochastic optimization question with bandit feedback and dynamic regret measures. The seminal work of Besbes et al. (2015) shows that, when aggregated function changes is known a priori, a simple re-starting algorithm attains the optimal dynamic regret. In this work, we designed a stochastic optimization algorithm with fixed step sizes, which combined together with the multi-scale sampling framework of Wei and Luo (2021) achieves the optimal dynamic regret in non-stationary stochastic optimization without requiring prior knowledge of function change b…
Non-Stationary Bandit Learning via Predictive Sampling
Jones Graduate School of Business, Rice University（里士满大学沃森商学院研究生院）；Stanford Graduate School of Business（斯坦福商学院）；Department of Management Science and Engineering, Department of Electrical Engineering, Stanford University（斯坦福大学管理科学与工程系、电气工程系）
University of Oslo（奥斯陆大学）；Toyota Technological Institute at Chicago（芝加哥丰田技术研究所）
We study the problem of non-stationary dueling bandits and provide the first adaptive dynamic regret algorithm for this problem. The only two existing attempts in this line of work fall short across multiple dimensions, including pessimistic measures of non-stationary complexity and non-adaptive parameter tuning that requires knowledge of the number of preference changes. We develop an elimination-based rescheduling algorithm to overcome these shortcomings and show a near-optimal $\tilde{O}(\sqrt{S^{\texttt{CW}} T})$ dynamic regret bound, where $S^{\texttt{CW}}$ is the number of times the Cond…
This paper tackles the risk averse multi-armed bandits problem when incurred losses are non-stationary. The conditional value-at-risk (CVaR) is used as the objective function. Two estimation methods are proposed for this objective function in the presence of non-stationary losses, one relying on a weighted empirical distribution of losses and another on the dual representation of the CVaR. Such estimates can then be embedded into classic arm selection methods such as epsilon-greedy policies. Simulation experiments assess the performance of the arm selection algorithms based on the two novel es…
Dynamic Regret for Non-Stationary Linear Bandits via Misspecification Reductions
Department of Mathematics, The Hong Kong University of Science and Technology（香港科技大学数学系）；Stern School of Business, New York University（纽约大学斯特恩商学院）；Department of IEDA, The Hong Kong University of Science and Technology（香港科技大学工业工程与决策分析系）
We consider the non-stationary multi-armed bandit (MAB) framework and propose a Kolmogorov-Smirnov (KS) test based Thompson Sampling (TS) algorithm named TS-KS, that actively detects change points and resets the TS parameters once a change is detected. In particular, for the two-armed bandit case, we derive bounds on the number of samples of the reward distribution to detect the change once it occurs. Consequently, we show that the proposed algorithm has sub-linear regret. Contrary to existing works, our algorithm is able to detect a change when the underlying reward distribution changes even …
We study the non-stationary stochastic multi-armed bandit problem, where the reward statistics of each arm may change several times during the course of learning. The performance of a learning algorithm is evaluated in terms of their dynamic regret, which is defined as the difference between the expected cumulative reward of an agent choosing the optimal arm in every time step and the cumulative reward of the learning algorithm. One way to measure the hardness of such environments is to consider how many times the identity of the optimal arm can change. We propose a method that achieves, in $K…
We consider a Multi-Armed Bandit problem in which the rewards are non-stationary and are dependent on past actions and potentially on past contexts. At the heart of our method, we employ a recurrent neural network, which models these sequences. In order to balance between exploration and exploitation, we present an energy minimization term that prevents the neural network from becoming too confident in support of a certain action. This term provably limits the gap between the maximal and minimal probabilities assigned by the network. In a diverse set of experiments, we demonstrate that our met…
A Decentralized Policy with Logarithmic Regret for a Class of Multi-Agent Multi-Armed Bandit Problems with Option Unavailability Constraints and Stochastic Communication Protocols
Sri Lanka Technological Campus（斯里兰卡科技园区）；University of Peradeniya（佩拉德尼亚大学）
A New Algorithm for Non-stationary Contextual Bandits: Efficient, Optimal, and Parameter-free
University of Southern California（南加州大学）
我们提出了第一个上下文强盗算法，是无参数，高效，最佳的动态遗憾。具体来说，我们的算法实现了动态遗憾$\mathcal{O}(\min\{\sqrt{ST}, Δ^{\frac{1}{3}}T^{\frac{2}{3}}\})$的上下文强盗问题与$T$轮，$S$开关和$Δ$总变化的数据分布。重要的是，我们的算法是自适应的，不需要提前知道$S$或$Δ$，并可以有效地实现假设访问一个ERM oracle。我们的结果严格地改进了（Luo et al.，2018），并大大推广和改进了（Auer等人，2018）的$\mathcal{O}(\sqrt{ST})$结果，该结果仅适用于没有上下文信息的双臂强盗问题。我们的算法的关键新颖性是引入重放阶段，在该阶段中，算法根据其先前的决定在一定的时间内采取行动，以检测非平稳性，同时保持探索和开发之间的良好平衡。
The stochastic multi-armed bandit has provided a framework for studying decision-making in unknown environments. We propose a variant of the stochastic multi-armed bandit where the rewards are sampled from a stochastic linear dynamical system. The proposed strategy for this stochastic multi-armed bandit variant is to learn a model of the dynamical system while choosing the optimal action based on the learned model. Motivated by mathematical finance areas such as Intertemporal Capital Asset Pricing Model proposed by Merton and Stochastic Portfolio Theory proposed by Fernholz that both model ass…
Best Arm Identification in Generalized Linear Bandits
Stanford University（斯坦福大学）
受药物设计的启发，我们考虑了广义线性Bandits中的最佳臂识别问题。更具体地说，我们假设每个手臂都有一个协变量向量，有一个未知的参数向量，这些参数向量在手臂之间是共同的，并且广义线性模型捕获了奖励对协变量和参数向量的依赖性。问题是最小化识别具有足够高概率的足够接近最优的臂所需的臂拉动的数量。基于线性强盗最佳臂识别的最新进展（Xu et al. 2018），我们提出了第一个广义线性强盗最佳臂识别算法，为其准确性和采样效率提供了理论保证，并通过仿真评估其在各种场景下的性能。
UC San Diego（加州大学圣地亚哥分校）；AWS AI（亚马逊云科技人工智能）；UC Berkeley（加州大学伯克利分校）
Understanding complex dynamics of two-sided online matching markets, where the demand-side agents compete to match with the supply-side (arms), has recently received substantial interest. To that end, in this paper, we introduce the framework of decentralized two-sided matching market under non stationary (dynamic) environments. We adhere to the serial dictatorship setting, where the demand-side agents have unknown and different preferences over the supply-side (arms), but the arms have fixed and known preference over the agents. We propose and analyze a decentralized and asynchronous learning…
