SHADHO：大规模可扩展的硬件感知分布式超参数优化
SHADHO: Massively Scalable Hardware-Aware Distributed Hyperparameter Optimization
摘要：
计算机视觉正在经历人工智能的复兴，其中机器学习模型正在加速学术研究和商业应用的重要突破。然而，有效地训练这些模型并不容易，部分原因是超参数：用户配置的值控制模型从数据中学习的能力。现有的超参数优化方法是高度并行的，但没有努力平衡跨异构硬件的搜索或优先搜索高影响空间。在本文中，我们介绍了一个大规模可扩展的硬件感知分布式超参数优化（SHADHO）的框架。我们的框架计算每个搜索空间的相对复杂度，并在所有试验中监控学习任务的性能。然后，这些指标被用作算法，以基于其硬件将超参数分配给分布式工作者。我们首先证明，我们的框架实现了一倍的吞吐量的标准分布式超参数优化框架优化SVM MNIST使用150个分布式工人。然后，我们在一周的时间里使用两个计算集群中的74个GPU使用SHADHO进行模型搜索，以优化U-Net的细胞分割任务，发现515个模型实现了比标准U-Net更低的验证损失。
英文摘要：
Computer vision is experiencing an AI renaissance, in which machine learning models are expediting important breakthroughs in academic research and commercial applications. Effectively training these models, however, is not trivial due in part to hyperparameters: user-configured values that control a model's ability to learn from data. Existing hyperparameter optimization methods are highly parallel but make no effort to balance the search across heterogeneous hardware or to prioritize searching high-impact spaces. In this paper, we introduce a framework for massively Scalable Hardware-Aware Distributed Hyperparameter Optimization (SHADHO). Our framework calculates the relative complexity of each search space and monitors performance on the learning task over all trials. These metrics are then used as heuristics to assign hyperparameters to distributed workers based on their hardware. We first demonstrate that our framework achieves double the throughput of a standard distributed hyperparameter optimization framework by optimizing SVM for MNIST using 150 distributed workers. We then conduct model search with SHADHO over the course of one week using 74 GPUs across two compute clusters to optimize U-Net for a cell segmentation task, discovering 515 models that achieve a lower validation loss than standard U-Net.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
