arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2206.03019cs.LGstat.ML
收藏

生存赌博机问题

The Survival Bandit Problem

The University of Tokyo(东京大学);RIKEN Center for AIP(理化学研究所革新智能统合研究中心);Kyoto University(京都大学)

Charles Riou, Junya Honda, Masashi Sugiyama

AI总结:

本文提出多臂赌博机新变体生存赌博机问题(S-MAB),在累积奖励低于阈值时中断程序,通过帕累托意义最小化生存遗憾并推导破产概率下界匹配策略,解决了Perotto等人(COLT 2019)的开放问题。

摘要:

我们介绍并研究了一种多臂赌博机问题(MAB)的新变体,称为生存赌博机问题(S-MAB)。在这两个问题中,目标都是最大化所谓的累积奖励,但在这一新变体中,如果累积奖励低于预设阈值,程序将被中断。这种简单但尚未被探索的 MAB 扩展源于许多实际应用。例如,在志愿患者身上测试两种药物的对比效果时,人们的健康面临风险,因此如果出现严重副作用或治疗未能消除疾病症状,必须能够中断实验。从理论角度来看,S-MAB 是 MAB 的第一个程序可能被中断也可能不被中断的变体。我们首先对 S-MAB 进行形式化,并将其目标定义为最小化所谓的生存遗憾,这自然地推广了 MAB 的遗憾。然后,我们展示了 S-MAB 的目标比 MAB 困难得多,因为与 MAB 相反,没有任何策略能够实现合理小的(即次线性的)生存遗憾。相反,我们在帕累托意义上最小化生存遗憾,即我们寻找一种策略,其累积奖励无法在不牺牲某个问题实例的情况下为另一个实例得到改进。为此,我们在生存遗憾中识别出两个关键组成部分:无破产条件下的遗憾(对应于 MAB 中的遗憾),以及程序被中断的概率,称为破产概率。我们推导了破产概率的下界,并提出了破产概率与该下界相匹配的策略。最后,基于这些策略的倍增技巧,我们推导出了一种在帕累托意义上最小化生存遗憾的策略,从而回答了 Perotto 等人(COLT 2019)提出的一个开放问题。

英文摘要:

We introduce and study a new variant of the multi-armed bandit problem (MAB), called the survival bandit problem (S-MAB). While in both problems, the objective is to maximize the so-called cumulative reward, in this new variant, the procedure is interrupted if the cumulative reward falls below a preset threshold. This simple yet unexplored extension of the MAB follows from many practical applications. For example, when testing two medicines against each other on voluntary patients, people's health are at stake, and it is necessary to be able to interrupt experiments if serious side effects occur or if the disease syndromes are not dissipated by the treatment. From a theoretical perspective, the S-MAB is the first variant of the MAB where the procedure may or may not be interrupted. We start by formalizing the S-MAB and we define its objective as the minimization of the so-called survival regret, which naturally generalizes the regret of the MAB. Then, we show that the objective of the S-MAB is considerably more difficult than the MAB, in the sense that contrary to the MAB, no policy can achieve a reasonably small (i.e., sublinear) survival regret. Instead, we minimize the survival regret in the sense of Pareto, i.e., we seek a policy whose cumulative reward cannot be improved for some problem instance without being sacrificed for another one. For that purpose, we identify two key components in the survival regret: the regret given no ruin (which corresponds to the regret in the MAB), and the probability that the procedure is interrupted, called the probability of ruin. We derive a lower bound on the probability of ruin, as well as policies whose probability of ruin matches the lower bound. Finally, based on a doubling trick on those policies, we derive a policy which minimizes the survival regret in the sense of Pareto, giving an answer to an open problem by Perotto et al. (COLT 2019).

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑