香草策略优化在随机上下文赌博机中既最优又具有差分隐私
Vanilla Policy Optimization Is Both Optimal and Differentially Private for Stochastic Contextual Bandits
- University of Illinois Chicago（伊利诺伊大学芝加哥分校）
- Toyota Technological Institute at Chicago（芝加哥丰田技术研究所）
- Tel Aviv University（特拉维夫大学）
- University of Toronto（多伦多大学）
- Vector Institute（向量研究所）
- Google Research（谷歌研究院）
机构由 AI 辅助整理，请以论文原文为准。
AI总结：
本文证明香草策略优化在随机上下文赌博机中无需探索奖励即可实现接近最优遗憾，并扩展出差分隐私版本，通过隐式探索机制和批量更新降低计算成本，实验验证其有效性。
AI中文摘要：
香草策略优化能否在随机上下文赌博机中充分探索以实现接近最优的遗憾？我们证明，在可实现性条件下，由离线回归驱动的标准指数策略更新能够做到这一点，而无需探索奖励或重要性加权。对于$A$个动作、$T$轮以及有限预测类$F$，香草策略优化以高概率实现$\tilde O(\sqrt{AT\log(|F|)})$的遗憾。我们的分析揭示了一种具有独立价值的隐式探索机制：渐进式策略更新防止动作过快失去概率，从而允许回归预言机学习其期望损失。我们进一步开发了一个批量版本，仅使用$O(\log T)$次回归调用和策略切换，并展示了私有回归预言机如何在不跨批次组合的情况下产生差分隐私的上下文赌博机算法。对于有限类，这给出了纯$\varepsilon_{\rm priv}$-DP和遗憾$\tilde O\left( \sqrt{AT \log(|F|/\delta)}(1+\varepsilon_{\rm priv}^{-1/2}) \right)$。最后，在基于预言机的上下文赌博机算法上的实验（包括有隐私和无隐私情况）证明了策略优化的实际有效性，以及在更强隐私约束下显式探索的价值。
英文摘要：
Can vanilla policy optimization explore enough to achieve near-optimal regret in stochastic contextual bandits? We show that standard exponential policy updates driven by offline regression do so under realizability, without exploration bonuses or importance weighting. For $A$ actions, $T$ rounds, and a finite prediction class $F$, vanilla PO achieves $\widetilde O(\sqrt{AT\log(|F|)})$ regret with high probability. Our analysis reveals an implicit exploration mechanism of independent interest: gradual policy updates prevent actions from losing probability too quickly, allowing the regression oracle to learn their expected losses. We further develop a batched version using only $O(\log T)$ regression calls and policy switches, and show how private regression oracles yield differentially private contextual bandit algorithms without composition across batches. For a finite class, this gives pure $\varepsilon_{\rm priv}$-DP and regret $\widetilde O\left( \sqrt{AT \log(|F|/δ)}(1+\varepsilon_{\rm priv}^{-1/2}) \right)$. Finally, experiments across oracle-based contextual bandit algorithms, with and without privacy, demonstrate the practical effectiveness of policy optimization and the value of explicit exploration under stronger privacy constraints.
