arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2610.19804cs.LGstat.ML
收藏

基于通用函数近似的高效上下文部分监控

Efficient contextual partial monitoring with general function approximation

University of Arizona(亚利桑那大学);POSTECH(浦项科技大学)

Kapilan Balagopalan, Hao Qin, Kwang-Sung Jun, Chicheng Zhang

AI总结:

该研究针对带侧信息的有限随机部分监控博弈,基于E2D元算法设计高效框架,提出两种探索规则,在合成博弈与LLM路由任务上验证其性能优于现有方法。

摘要:

我们研究带有侧信息的有限随机部分监控(PM)博弈,这是一种带有部分反馈的序列学习范式。在每一轮 $t \in [T]$ 中,玩家和环境都会观察一个上下文,玩家选择一个动作,环境同时选择一个结果。玩家既无法观察到结果,也无法观察到损失,只能观察到一个对结果具有部分信息的反馈符号。我们利用Foster等人(2021)提出的估计到决策(E2D)元算法,设计了一种用于上下文部分监控的计算高效框架,该框架使用在线回归预言机,基于历史(上下文、动作、反馈符号)元组高效估计结果分布。由于原始E2D算法给出的探索规则难以计算,我们提出两种计算高效的探索规则,可实现最优的 $T$ 阶遗憾:在全局可观测设置中,我们的探索规则基于 $\epsilon$-贪心;在局部可观测设置中,我们的探索规则使用逆间隙加权(Foster & Rakhlin, 2020),该方法此前仅在多臂赌博机设置中被证明有效。我们表明,名为该框架变体的算法可与离线回归预言机结合,当上下文从固定分布独立抽取时,能得到具有最优遗憾的高效算法。我们在合成的标签高效预测博弈和大语言模型(LLM)路由任务上评估了我们的算法,在该任务中,学习者根据提示、不确定性和内部状态特征,决定是否将查询从弱模型升级到强模型。结果显示,我们的方法相比现有方法具有更优性能。

英文摘要:

We study finite stochastic partial monitoring (PM) games with side information, a paradigm of sequential learning with partial feedback. On each round $t \in [T]$, both player and environment observe a context and player chooses an action and environment simultaneously chooses an outcome. The player neither observes the outcome nor the loss; instead, they only observe a feedback symbol which is partially informative about the outcome. We utilize the Estimation-to-Decisions (E2D) meta-algorithm of Foster et al.(2021) to design E2D.PMSide, a computationally efficient framework for contextual partial monitoring, which uses an online regression oracle to efficiently estimate the outcome distributions based on historical (context, action, feedback symbol) tuples. Since the exploration rule given by the original E2D algorithm is intractable to compute, we propose two computationally efficient exploration rules that achieve optimal regret orders in $T$: in the globally observable setting, our exploration rule is based on $ε$-greedy, while in the locally observable setting, our exploration rule uses inverse gap weighting (Foster & Rakhlin, 2020), which was previously only known to be effective in the bandit setting. We show that a variant of E2D.PMSide named OE2D.PMSide can be combined with offline regression oracles to give an efficient algorithm with optimal regret when the contexts are drawn independently from a fixed distribution. We evaluate our algorithm on a synthetic label-efficient prediction game and an LLM routing task, in which a learner decides from prompt, uncertainty, and internal state features whether to escalate a query from a weak model to a strong one. We show that our method has superior performance to prior approaches.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑