arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2610.30801cs.LG
收藏

带(聚合)臂式反馈的对抗线性马尔可夫决策过程的计算高效且速率最优算法

A Computationally Efficient and Rate-Optimal Algorithm for Adversarial Linear MDPs with (Aggregate) Bandit Feedback

University of Virginia(弗吉尼亚大学)

Haolin Liu, Chen-Yu Wei, Hanbin Zhou

AI总结:

本研究针对带聚合臂式反馈的对抗线性MDP,提出首个多项式时间算法,实现√K悔值,解决了现有算法计算低效或悔值次优的问题。

摘要:

我们研究具有未知转移、对抗损失和臂式反馈的分段线性马尔可夫决策过程(MDP)。该设置下的最新成果(Liu等人,2024)要么采用计算低效的算法实现√K的悔值(regret),要么实现次优的K^(3/4)悔值,其中K为分段数量。本研究提出首个采用多项式时间算法实现√K悔值的方案。我们的方法将对抗线性MDP归约为对抗线性臂式问题,其动作集由策略的期望特征向量构成。核心挑战在于该动作集未知,仅能通过探索进行估计。我们通过为动作集构建近似分离神谕(oracle)来解决这一挑战,该神谕要么从查询特征中生成可执行策略,要么优化动作集的外近似。该方法借鉴了van Erven等人(2025)针对线性上下文臂式问题的归约思路,以及Ito等人(2019)的神谕高效线性臂式算法,同时额外处理了神谕的不精确性。所得算法仅需聚合损失反馈,据我们所知,这是首个适用于超越表格型情况的、带聚合反馈的对抗MDP的多项式时间算法。

英文摘要:

We study episodic linear Markov decision processes with unknown transitions, adversarial losses, and bandit feedback. The state-of-the-art result for this setting (Liu et al., 2024) either achieves $\sqrt{K}$ regret with a computationally inefficient algorithm or achieves suboptimal $K^{3/4}$ regret, where $K$ is the number of episodes. Our work gives the first $\sqrt{K}$ regret with a polynomial-time algorithm. Our method reduces adversarial linear MDPs to adversarial linear bandits whose action set consists of policies' expected feature vectors. The central challenge is that this action set is unknown and can be estimated only through exploration. We address this challenge by constructing an approximate separation oracle for the action set that either generates executable policies from queried features, or refines an outer approximation of the action set. This approach draws inspiration from the reduction of van Erven et al. (2025) for linear contextual bandits and the oracle-efficient linear bandit algorithm of Ito et al. (2019), but additionally deals with the imprecision of the oracle. The resulting algorithm requires only aggregate loss feedback, and to our knowledge. This is the first polynomial-time algorithm for adversarial MDPs with aggregate feedback beyond the tabular case.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑