在无限时间MDP中计算和学习Berk-Nash均衡的数学规划方法
A Mathematical Programming Approach to Computing and Learning Berk--Nash Equilibria in Infinite-Horizon MDPs
AI总结：
本文研究了在内部模型类不准确时的序列决策问题，通过耦合线性规划和双层优化方法，提出了一种计算和学习Berk-Nash均衡的数学规划方法，并引入熵正则化以解决非光滑性问题，最终实现有效的探索-利用平衡。
摘要：
我们研究了当智能体的内部模型类不准确时的序列决策问题。在无限时间Berk-Nash框架中，稳定行为作为固定点出现：智能体相对于主观模型最优行动，同时该模型与由自身策略生成的长期数据统计一致。我们通过耦合线性规划和双层优化方法对这种均衡进行严格刻画。为解决标准最佳响应对应关系的内在非光滑性，我们引入熵正则化，建立唯一的软Bellman固定点和光滑目标。利用这种正则性，我们开发了一种在线学习方案，将模型选择转化为使用EXP3型更新的对抗性老虎机问题，并辅以一种新颖的猜测集缩放机制，以自适应地细化参数空间。数值结果展示了有效的探索-利用权衡、收敛到KL最小化模型以及子线性遗憾。
英文摘要：
We study sequential decision-making when the agent's internal model class is misspecified. Within the infinite-horizon Berk-Nash framework, stable behavior arises as a fixed point: the agent acts optimally relative to a subjective model, while that model is statistically consistent with the long-run data endogenously generated by the policy itself. We provide a rigorous characterization of this equilibrium via coupled linear programs and a bilevel optimization formulation. To address the intrinsic non-smoothness of standard best-response correspondences, we introduce entropy regularization, establishing the existence of a unique soft Bellman fixed point and a smooth objective. Exploiting this regularity, we develop an online learning scheme that casts model selection as an adversarial bandit problem using an EXP3-type update, augmented by a novel conjecture-set zooming mechanism that adaptively refines the parameter space. Numerical results demonstrate effective exploration-exploitation trade-offs, convergence to the KL-minimizing model, and sublinear regret.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
