We study the multi-armed bandit problem with arms which are Markov chains with rewards. In the finite-horizon setting, the celebrated Gittins indices do not apply, and the exact solution is intractable. We provide approximation algorithms for a more general model which includes Markov decision processes and non-unit transition times. When preemption is allowed, we provide a (1/2-eps)-approximation, along with an example showing this is tight. When preemption isn't allowed, we provide a 1/12-approximation, which improves to a 4/27-approximation when transition times are unity. Our model encompasses the Markovian Bandits model of Gupta et al, the Stochastic Knapsack model of Dean, Goemans, and Vondrak, and the Budgeted Learning model of Guha and Munagala, and our algorithms improve existing results in all three areas. In our analysis, we encounter and overcome to our knowledge a novel obstacle - an algorithm that provably exists via polyhedral arguments, but cannot be found in polynomial time.
背包强盗(Bandits with Knapsacks,BwK)是一种在供给/预算限制下的多武器强盗模型。虽然BwK的最坏情况下的遗憾界限是很好理解的,我们提出了三个结果,超出了最坏情况下的角度。首先,我们提供了一个完整的表征对数,实例依赖的后悔率的上限和下限。其次,我们考虑BwK中的“简单遗憾”,它跟踪算法在给定轮中的性能,并证明它在所有轮中都很小。第三,我们提供了一个一般的“减少”BwK土匪利用一些已知的有用的结构,并将这种减少组合半土匪,线性上下文土匪,和多项logit土匪。我们的结果建立在BwK算法从\citet{AgrawalDevanur-ec14},提供新的分析。
Non-monotonic Resource Utilization in the Bandits with Knapsacks Problem
Cornell University(康奈尔大学)
背包强盗(Bandits with Knapsacks,BwK)是一个有影响力的不确定性下的序贯决策模型,它包含了资源消耗约束。在每一轮中,决策者观察到一个由奖励和非负资源消耗向量组成的结果,每个资源的预算都随着其消耗而减少。在本文中,我们介绍了一个自然的推广的随机BwK问题,允许非单调的资源利用率。在每一轮中,决策者观察到一个由奖励和资源漂移向量组成的结果,资源漂移向量可以是正的、负的或零,每个资源的预算都随着其漂移而增加。我们的主要结果是一个马尔可夫决策过程(MDP)的政策,对线性规划(LP)放松时,决策者知道真正的结果分布有恒定的遗憾。在此基础上,我们开发了一种学习算法,当决策者不知道真实的结果分布时,该算法对相同的LP松弛具有对数遗憾。我们还提出了一个减少从BwK到我们的模型,表明我们的遗憾界匹配现有的结果。
Department of Statistics and Data Science, University of Pennsylvania(统计与数据科学系,宾夕法尼亚大学);Department of Mathematics, The Hong Kong University of Science and Technology(数学系,香港科学与技术大学);Department of Industrial Engineering & Decision Analytics, The Hong Kong University of Science and Technology(工业工程与决策分析系,香港科学与技术大学)
Adversarial Attacks on Combinatorial Multi-Armed Bandits
Oregon State University(俄勒冈州立大学);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Pennsylvania State University(宾夕法尼亚州立大学);Princeton University(普林斯顿大学)