平均奖励强化学习的有限常数前沿与可审计后悔证书

中文摘要 AI 辅助

平均奖励强化学习的后悔度仅在对数因子范围内已知，但已发表保证的数值内容难以比较，因为概率模式、结构参数、对数归一化、先验信息和规划假设存在差异。我们引入一种感知常数的比较协议，并为通信马尔可夫决策过程（MDP）推导显式有限下界证书。该构造是一个两状态块的二叉树，其证明使用精确的轨迹级伯努利KL散度，并保持动作预算、直径、占用率、导航成本和终端偏差的显式性。一个通用闭式包络在有限前沿上改进了已发表的系数0.015：在中等范围内为0.0200，在更强的动作、直径和视界条件下高达0.0291，增幅达94%。极限系数为$\frac{1}{32}\times\bigl(\frac{A-3}{A}\bigr)^{\frac{1}{2}}$。对于上界，我们给出了跨度约束的乐观学习者的可审计组合规则，但未声明系数，因为自适应方向方差和规划证书仍未解决。我们还形式化了有效的期望转换和常数可比性。受控诊断测试直径依赖性、宽度交互的奖励、跨度误 specification 以及有限下界证书在其精确族上的表现。

英文摘要

Average-reward reinforcement-learning regret is known up to logarithmic factors, but the numerical content of published guarantees is difficult to compare because probability mode, structural parameter, logarithmic normalization, prior information, and planning assumptions differ. We introduce a constant-aware comparison protocol and derive an explicit finite lower certificate for communicating MDPs. The construction is a binary tree of two-state blocks; its proof uses exact trajectory-level Bernoulli KL divergence and keeps action budget, diameter, occupancy, navigation cost, and terminal bias explicit. A common closed-form envelope improves the published coefficient $0.015$ across a finite frontier: $0.0200$ in a moderate regime and up to $0.0291$ under stronger action, diameter, and horizon conditions, a $94\%$ increase. The limiting coefficient is $\frac1{32}\sqrt{(A-3)/A}$. For upper bounds, we give an auditable composition rule for a span-constrained optimistic learner, but do not claim a coefficient while adaptive directional-variance and planning certificates remain open. We also formalize valid expectation conversion and constant comparability. Controlled diagnostics test diameter dependence, bonus-by-width interactions, span misspecification, and the finite lower certificate on its exact family.

发表机构

• Iowa State University（爱荷华州立大学）
• BRAC University（BRAC大学）

机构由 AI 辅助整理，请以论文原文为准。