arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2107.03263cs.LG
收藏

随机专家老虎机:常数遗憾、经验专家与回合

Bandits with Stochastic Experts: Constant Regret, Empirical Experts and Episodes

The University of Texas at Austin(德克萨斯大学奥斯汀分校);Google(谷歌公司);IBM Research(IBM研究院);Google DeepMind(谷歌DeepMind)

Nihal Sharma, Rajat Sen, Soumya Basu, Karthikeyan Shanmugam, Sanjay Shakkottai

AI总结:

针对随机专家上下文老虎机,提出D-UCB与仅需近似分布知识的ED-UCB,实现常数及回合变化环境下的遗憾保证并经模拟验证。

摘要:

我们研究上下文老虎机问题的一种变体,其中智能体可以通过一组随机专家策略进行干预。给定固定上下文,每位专家从固定条件分布中采样动作。智能体寻求与给定专家集合中的“最佳”专家保持竞争力。我们提出基于散度的上置信界(D-UCB)算法,该算法使用重要性采样在专家之间共享信息,并提供与视野无关的常数遗憾界,该界仅随专家数量线性增长。我们还提出经验D-UCB(ED-UCB)算法,它仅需专家分布的近似知识即可运行。此外,我们研究回合制设定,其中智能体与随回合变化的环境交互。每个回合可以具有不同的上下文和奖励分布,导致最佳专家随回合变化。我们表明,通过从 O(N log(NT^2 sqrt(E))) 个样本进行自举,对于 N 位专家、E 个长度均为 T 的回合,ED-UCB 保证遗憾按 O(E(N+1)+N sqrt(E)/T^2) 缩放。最后,我们通过模拟对研究结果进行实证验证。

英文摘要:

We study a variant of the contextual bandit problem where an agent can intervene through a set of stochastic expert policies. Given a fixed context, each expert samples actions from a fixed conditional distribution. The agent seeks to remain competitive with the 'best' among the given set of experts. We propose the Divergence-based Upper Confidence Bound (D-UCB) algorithm that uses importance sampling to share information across experts and provide horizon-independent constant regret bounds that only scale linearly in the number of experts. We also provide the Empirical D-UCB (ED-UCB) algorithm that can function with only approximate knowledge of expert distributions. Further, we investigate the episodic setting where the agent interacts with an environment that changes over episodes. Each episode can have different context and reward distributions resulting in the best expert changing across episodes. We show that by bootstrapping from $\mathcal{O}\left(N\log\left(NT^2\sqrt{E}\right)\right)$ samples, ED-UCB guarantees a regret that scales as $\mathcal{O}\left(E(N+1) + \frac{N\sqrt{E}}{T^2}\right)$ for $N$ experts over $E$ episodes, each of length $T$. We finally empirically validate our findings through simulations.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑