The landmark achievements of AlphaGo Zero have created great research interest into self-play in reinforcement learning. In self-play, Monte Carlo Tree Search is used to train a deep neural network, that is then used in tree searches. Training itself is governed by many hyperparameters.There has been surprisingly little research on design choices for hyper-parameter values and loss-functions, presumably because of the prohibitive computational cost to explore the parameter space. In this paper, we investigate 12 hyper-parameters in an AlphaZero-like self-play algorithm and evaluate how these parameters contribute to training. We use small games, to achieve meaningful exploration with moderate computational effort. The experimental results show that training is highly sensitive to hyper-parameter choices. Through multi-objective analysis we identify 4 important hyper-parameters to further assess. To start, we find surprising results where too much training can sometimes lead to lower performance. Our main result is that the number of self-play iterations subsumes MCTS-search simulations, game-episodes, and training epochs. The intuition is that these three increase together as self-play iterations increase, and that increasing them individually is sub-optimal. A consequence of our experiments is a direct recommendation for setting hyper-parameter values in self-play: the overarching outer-loop of self-play iterations should be maximized, in favor of the three inner-loop hyper-parameters, which should be set at lower values. A secondary result of our experiments concerns the choice of optimization goals, for which we also provide recommendations.
A Trajectory-Based Bayesian Approach to Multi-Objective Hyperparameter Optimization with Epoch-Aware Trade-Offs
Department of Industrial Systems Engineering and Management, National University of Singapore（新加坡国立大学工业系统工程与管理系）；Academy of Mathematics and System Sciences, Chinese Academy of Sciences（中国科学院数学系统科学 academy）；School of Mathematical Sciences, University of Chinese Academy of Sciences（中国科学院大学数学科学学院）
The Unreasonable Effectiveness Of Early Discarding After One Epoch In Neural Network Hyperparameter Optimization
Argonne National Laboratory（阿贡国家实验室）；Université Paris-Saclay（巴黎-萨克雷大学）；Universidad de La Sabana（萨瓦纳大学）；Delft University of Technology（代尔夫特理工大学）；Oak Ridge National Laboratory（橡树岭国家实验室）
HyperFuse: Fast Self-Supervised Node Embeddings for Attributed Hypergraphs
Indian Institute of Science Education and Research Thiruvananthapuram（印度科学教育与研究学院蒂鲁文南特普拉姆分校）；Indian Institute of Science Education and Research Kolkata（印度科学教育与研究学院加尔各答分校）；University of Rostock（罗斯托克大学）；Stellenbosch Institute for Advanced Study (STIAS)（斯泰伦博斯高等研究院）
mHC-lite: You Don't Need 20 Sinkhorn-Knopp Iterations
CSE, University of Michigan, Ann Arbor, USA（计算机科学与工程系，密歇根大学，安娜堡，美国）；CCDS, Nanyang Technological University, Singapore（计算机科学与数据科学系，南洋理工大学，新加坡）；CBS-NTT Physics of Intelligence Program, Harvard University, Cambridge, USA（哈佛大学剑桥分校人工智能物理计划）；Physics of Artificial Intelligence Group, NTT Research Inc, USA（人工智能物理研究组，NTT研究公司，美国）
Optimization-Derived Learning with Essential Convergence Analysis of Training and Hyper-training
Dalian University of Technology（大连理工大学）；Peng Cheng Laboratory（鹏城实验室）；University of Victoria（维多利亚大学）；Southern University of Science and Technology（南方科技大学）；National Center for Applied Mathematics Shenzhen（深圳国家应用数学中心）
