加权计数老虎机：通过重复暴露最优性克服难解性
Weighted Tallying Bandits: Overcoming Intractability via Repeated Exposure Optimality
AI总结：
针对人类记忆衰减下的在线学习动作损失建模问题，提出加权计数老虎机设定，在重复暴露最优性条件下改进连续消除算法，实现近乎最优的完全策略遗憾保证，且计算高效、实验表现优于基线。
摘要：
在在线学习的推荐系统或众包应用中，人类的偏好或能力通常是算法近期动作的函数。受此启发，大量研究工作将一类场景形式化：动作的损失是该动作在之前$m$个时间步中被执行次数的函数，其中$m$对应人类记忆容量的上限。为了更真实地捕捉人类记忆随时间的衰减特性，我们提出了加权计数老虎机（Weighted Tallying Bandit，WTB），该设定对上述场景进行了推广，要求动作的损失是该臂在过去$m$个时间步中被执行次数的加权求和的函数。若不引入额外假设，WTB设定是难解的。因此我们在重复暴露最优性（Repeated Exposure Optimality，REO）条件下对其展开研究，该条件受人类生理学文献启发，要求存在一个动作，当被重复执行时，最终会产生比任何其他动作序列都更小的损失。我们研究了REO条件下WTB中的完全策略遗憾（complete policy regret，CPR）最小化问题，CPR是最强的遗憾概念。由于$m$通常是未知的，我们假设仅能获取$m$的上界$M$。我们证明，对于具有$K$个动作、时间跨度为$T$的问题，对连续消除算法进行简单修改后，其CPR为$O \left( \sqrt{KT} + (m+M)K \right)$。有趣的是，除了$(m+M)K$的加性因子（而非乘性因子）外，该结果恢复了更简单的随机多臂老虎机在传统遗憾下的经典保证。我们还证明，在我们的设定中，任何算法的加性CPR都为$Ω\left( mK + M \right)$，这表明我们的结果几乎是最优的。我们的算法计算效率高，并且我们通过实验证明了其实用性以及相对于常见基线的优越性。
英文摘要：
In recommender system or crowdsourcing applications of online learning, a human's preferences or abilities are often a function of the algorithm's recent actions. Motivated by this, a significant line of work has formalized settings where an action's loss is a function of the number of times that action was recently played in the prior $m$ timesteps, where $m$ corresponds to a bound on human memory capacity. To more faithfully capture decay of human memory with time, we introduce the Weighted Tallying Bandit (WTB), which generalizes this setting by requiring that an action's loss is a function of a \emph{weighted} summation of the number of times that arm was played in the last $m$ timesteps. This WTB setting is intractable without further assumption. So we study it under Repeated Exposure Optimality (REO), a condition motivated by the literature on human physiology, which requires the existence of an action that when repetitively played will eventually yield smaller loss than any other sequence of actions. We study the minimization of the complete policy regret (CPR), which is the strongest notion of regret, in WTB under REO. Since $m$ is typically unknown, we assume we only have access to an upper bound $M$ on $m$. We show that for problems with $K$ actions and horizon $T$, a simple modification of the successive elimination algorithm has $O \left( \sqrt{KT} + (m+M)K \right)$ CPR. Interestingly, upto an additive (in lieu of mutliplicative) factor in $(m+M)K$, this recovers the classical guarantee for the simpler stochastic multi-armed bandit with traditional regret. We additionally show that in our setting, any algorithm will suffer additive CPR of $Ω\left( mK + M \right)$, demonstrating our result is nearly optimal. Our algorithm is computationally efficient, and we experimentally demonstrate its practicality and superiority over natural baselines.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
