Yahtzee：用于随机组合游戏的强化学习技术
Yahtzee: Reinforcement Learning Techniques for Stochastic Combinatorial Games
AI总结：
本文提出使用强化学习技术解决随机组合游戏Yahtzee的多人策略问题，通过A2C方法在有限预算下达到接近最优性能，但其他方法在超参数敏感性和长期信用分配方面存在挑战。
摘要：
Yahtzee是一种具有随机性和组合结构的经典骰子游戏，具有延迟奖励，使其成为一个有趣的中等规模强化学习基准。虽然使用动态规划方法可以计算出单人Yahtzee的最优策略，但多人游戏则难以处理，促使了近似方法的出现。我们将Yahtzee建模为马尔可夫决策过程（MDP），并使用各种策略梯度方法训练自我对战的智能体：REINFORCE、优势演员评论家（A2C）和近端策略优化（PPO），均使用多头网络与共享主干。我们通过消融特征和动作编码、架构、回报估计器和熵正则化来理解其对学习的影响。在固定的训练预算下，REINFORCE和PPO对超参数敏感，无法达到接近最优的性能，而A2C则在各种设置中训练稳健。我们的智能体在100,000次评估游戏中获得241.78分，接近最优动态规划得分254.59分的5.0%，实现了上部奖金和Yahtzee的比率分别为24.9%和34.1%。所有模型都难以学习上部奖金策略，过度关注四条一的，突显了长期信用分配和探索的持续挑战。
英文摘要：
Yahtzee is a classic dice game with a stochastic, combinatorial structure and delayed rewards, making it an interesting mid-scale RL benchmark. While an optimal policy for solitaire Yahtzee can be computed using dynamic programming methods, multiplayer is intractable, motivating approximation methods. We formulate Yahtzee as a Markov Decision Process (MDP), and train self-play agents using various policy gradient methods: REINFORCE, Advantage Actor-Critic (A2C), and Proximal Policy Optimization (PPO), all using a multi-headed network with a shared trunk. We ablate feature and action encodings, architecture, return estimators, and entropy regularization to understand their impact on learning. Under a fixed training budget, REINFORCE and PPO prove sensitive to hyperparameters and fail to reach near-optimal performance, whereas A2C trains robustly across a range of settings. Our agent attains a median score of 241.78 points over 100,000 evaluation games, within 5.0\% of the optimal DP score of 254.59, achieving the upper section bonus and Yahtzee at rates of 24.9\% and 34.1\%, respectively. All models struggle to learn the upper bonus strategy, overindexing on four-of-a-kind's, highlighting persistent long-horizon credit-assignment and exploration challenges.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
