arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.08041cs.LGcs.AIcs.CL

隐式策略优化：重新思考对抗扑克环境中的长 horizon 决策

Implicit Strategic Optimization: Rethinking Long-Horizon Decision-Making in Adversarial Poker Environments

Gradient

Boyang Xia, Weiyou Tian, Qingnan Ren, Jiaqi Huang, Jie Xiao, Shuo Lu, Kai Wang, Lynn Ai, Eric Yang, Bill Shi

阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出隐式策略优化（ISO）框架，通过预测战略环境和结合战略奖励模型与乐观学习规则，提升对抗性扑克环境中的长 horizon 决策性能。

摘要：

训练大型语言模型（LLM）代理进行对抗性游戏通常由回合制目标（如胜率）驱动。然而在长 horizon 设置中，收益受到潜伏的战略外部性影响，这些外部性随时间演变，因此短视优化和基于变异的遗憾分析即使在动态可预测时也可能变得无效。为了解决这个问题，我们引入了隐式策略优化（ISO），这是一种预测意识框架，其中每个代理预测当前的战略环境并使用它来在线更新其策略。ISO结合了一个战略奖励模型（SRM），该模型估计动作的长期战略价值，以及iso-grpo，一种基于上下文的乐观学习规则。我们证明了子线性上下文遗憾和均衡收敛保证，其主导项与上下文误预测的数量成比例；当预测误差被限制时，我们的界限恢复了当战略外部性已知时获得的静态游戏速率。在6人无限制德州扑克和竞争型宝可梦中的实验显示，ISO在长期回报上优于强大的LLM和RL基线，并在受控的预测噪声下表现出良好的退化性能。

英文摘要：

Training large language model (LLM) agents for adversarial games is often driven by episodic objectives such as win rate. In long-horizon settings, however, payoffs are shaped by latent strategic externalities that evolve over time, so myopic optimization and variation-based regret analyses can become vacuous even when the dynamics are predictable. To solve this problem, we introduce Implicit Strategic Optimization (ISO), a prediction-aware framework in which each agent forecasts the current strategic context and uses it to update its policy online. ISO combines a Strategic Reward Model (SRM) that estimates the long-run strategic value of actions with iso-grpo, a context-conditioned optimistic learning rule. We prove sublinear contextual regret and equilibrium convergence guarantees whose dominant terms scale with the number of context mispredictions; when prediction errors are bounded, our bounds recover the static-game rates obtained when strategic externalities are known. Experiments in 6-player No-Limit Texas Hold'em and competitive Pokemon show consistent improvements in long-term return over strong LLM and RL baselines, and graceful degradation under controlled prediction noise.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑