We study finite stochastic partial monitoring (PM) games with side information, a paradigm of sequential learning with partial feedback. On each round $t \in [T]$, both player and environment observe a context and player chooses an action and environment simultaneously chooses an outcome. The player neither observes the outcome nor the loss; instead, they only observe a feedback symbol which is partially informative about the outcome. We utilize the Estimation-to-Decisions (E2D) meta-algorithm of Foster et al.(2021) to design E2D.PMSide, a computationally efficient framework for contextual partial monitoring, which uses an online regression oracle to efficiently estimate the outcome distributions based on historical (context, action, feedback symbol) tuples. Since the exploration rule given by the original E2D algorithm is intractable to compute, we propose two computationally efficient exploration rules that achieve optimal regret orders in $T$: in the globally observable setting, our exploration rule is based on $ε$-greedy, while in the locally observable setting, our exploration rule uses inverse gap weighting (Foster & Rakhlin, 2020), which was previously only known to be effective in the bandit setting. We show that a variant of E2D.PMSide named OE2D.PMSide can be combined with offline regression oracles to give an efficient algorithm with optimal regret when the contexts are drawn independently from a fixed distribution. We evaluate our algorithm on a synthetic label-efficient prediction game and an LLM routing task, in which a learner decides from prompt, uncertainty, and internal state features whether to escalate a query from a weak model to a strong one. We show that our method has superior performance to prior approaches.
Regret Lower Bound and Optimal Algorithm in Finite Stochastic Partial Monitoring
The University of Tokyo(东京大学)
部分监控是一个一般模型的顺序学习与有限的反馈形式化为两个球员之间的游戏。在这个游戏中,学习者选择一个动作,同时对手选择一个结果,然后学习者遭受损失,并收到反馈信号。学习者的目标是使总损失最小化。在本文中,我们研究了有限行动和随机结果的部分监控。我们推导出一个对数分布相关的遗憾下限,定义了问题的难度。受DMED算法(Honda and Takemura,2010)的启发,我们提出了PM-DMED算法,该算法最小化了分布相关的后悔。PM-DMED在数值实验中的性能明显优于最先进的算法。为了证明PM-DMED在遗憾界上的最优性,我们通过引入铰链函数(PM-DMED-Hinge)对算法进行了稍微修改。然后,我们得到了一个渐进最优的后悔PM-DMED-铰链的上界匹配的下限。
Decision-Focused On-Policy Learning for Contextual Linear Optimization with Partial Feedback
Department of Industrial Engineering and Operations Research, University of California, Berkeley(工业工程与运筹学系,加州大学伯克利分校);H. Milton Stewart School of Industrial and Systems Engineering, Georgia Institute of Technology(H.米尔顿·斯图尔特工业与系统工程学院,佐治亚理工学院)
The on-line shortest path problem under partial monitoring
Computer and Automation Research Institute of the Hungarian Academy of Sciences(匈牙利科学院计算机与自动化研究所);Queen’s University(女王大学);ICREA(加泰罗尼亚研究与高级研究院);Universitat Pompeu Fabra(庞培法布拉大学);Budapest University of Technology and Economics(布达佩斯技术与经济大学)
Neural Active Learning Meets the Partial Monitoring Framework
Université Laval(拉瓦尔大学);Thales Research and Technology (cortAIx)(泰勒斯研究与技术(cortAIx));Mila - Québec AI Institute(米拉-魁北克人工智能研究所);Institut Intelligence et Données(智能与数据研究所)
Robust approachability and regret minimization in games with partial monitoring
Israel Institute of Technology (Technion)(以色列理工学院);Université Paris-Diderot(巴黎第七大学(巴黎狄德罗大学));Ecole Normale Supérieure(巴黎高等师范学院);CNRS(法国国家科学研究中心);INRIA(法国国家信息与自动化研究所);HEC Paris(巴黎高等商学院)
在完全监控的可逼近性中,已知有两种类型的条件对于凸集是等价的:原始条件和对偶条件。原始的形式是:一个集合C是可接近的当且仅当所有包含的半空间在一次博弈中是可接近的;而对偶的形式是:一个凸集C是可接近的当且仅当它与所有具有某种形式的支付集相交。我们考虑部分监控博弈中的可接近性。在以前的工作(Perchet 2011; Mannor et al. 2011)中,我们提供了可逼近凸集的对偶特征;我们还展示了C是多胞形的情况下的有效策略。本文给出了凸集可部分监控逼近的基本条件。它们依赖于修改后的报酬函数,并导致可接近性策略,基于修改后的支付函数,进行类似于Blackwell(1956)的策略的预测;这是在对比与以前研究的策略,在这种情况下,主要依赖于信号结构,旨在估计以及所收到的信号的分布。我们的结果推广了Kohlberg 1975的经典结果(也可参见Mertens et al. 1994),并适用于具有任意信号结构的博弈以及任意凸集。
Instance-Dependent Regret Bounds for Nonstochastic Linear Partial Monitoring
EPFL(苏黎世联邦理工学院);Università degli Studi di Milano(米兰大学);Politecnico di Milano(米兰理工大学);ETH Zürich(苏黎世联邦理工学院);Univ. Grenoble Alpes(格勒诺布尔阿尔卑斯大学);Inria(法国国家信息与自动化技术研究院);CNRS(法国国家科学研究中心);Grenoble INP(格勒诺布尔INP);LJK(实验室)
Algorithmic Chaining and the Role of Partial Feedback in Online Nonparametric Learning
Università degli Studi di Milano(米兰大学);INRIA(法国国家信息与自动化研究所);École Normale Supérieure(巴黎高等师范学院);Università degli Studi dell’Insubria(因苏布里亚大学);Université Toulouse III - Paul Sabatier(图卢兹第三大学(保罗·萨巴蒂埃大学))