While selecting the hyper-parameters of Neural Networks (NNs) has been so far treated as an art, the emergence of more complex, deeper architectures poses increasingly more challenges to designers and Machine Learning (ML) practitioners, especially when power and memory constraints need to be considered. In this work, we propose HyperPower, a framework that enables efficient Bayesian optimization and random search in the context of power- and memory-constrained hyper-parameter optimization for NNs running on a given hardware platform. HyperPower is the first work (i) to show that power consumption can be used as a low-cost, a priori known constraint, and (ii) to propose predictive models for the power and memory of NNs executing on GPUs. Thanks to HyperPower, the number of function evaluations and the best test error achieved by a constraint-unaware method are reached up to 112.99x and 30.12x faster, respectively, while never considering invalid configurations. HyperPower significantly speeds up the hyper-parameter optimization, achieving up to 57.20x more function evaluations compared to constraint-unaware methods for a given time interval, effectively yielding significant accuracy improvements by up to 67.6%.
National Special Education Resource Center for Children with Autism, Zhejiang Normal University(浙江师范大学国家孤独症儿童特殊教育资源中心);Institute of Cyber-Systems and Control, Zhejiang University(浙江大学 cyber 系统与控制研究所);School of Computer Science and Technology, Zhejiang Normal University(浙江师范大学计算机科学与技术学院)
Accelerating Birkhoff Projection for Manifold-Constrained Hyper-Connections
School of Statistics(统计学系);Renmin University of China(中国人民大学);School of Statistics and Data Science(统计学与数据科学学院);Institute of Big Data Research(大数据研究院);Shanghai University of Finance and Economics(上海财经大学)
Tuning Word2vec for Large Scale Recommendation Systems
Twitter(推特);Imperial College London(伦敦帝国理工学院)
Word2vec是一种功能强大的机器学习工具,它起源于自然语言处理(NLP),目前已应用于推荐系统、预测和网络分析等多个领域。由于Word2vec经常使用现成的,我们解决的问题,是否默认的超参数是适合的推荐系统。在本文中,我们首先阐明了超参数优化的重要性,并表明,无约束优化产生了平均221%的提高命中率比thefdefault参数。然而,无约束优化会导致超参数设置非常昂贵,并且对于大规模优化任务不可行。为此,我们证明了138%的平均改善命中率与运行时间约束的超参数优化。此外,为了使超参数优化适用于大规模推荐问题,其中目标数据集太大,无法搜索,我们研究了从样本中泛化超参数设置。我们表明,应用约束超参数优化,仅使用10%的数据样本仍然产生了91%的平均改善命中率超过默认参数时,应用到thefuldatasets。最后,我们将使用我们的样本约束优化方法学习的超参数应用于Twitter上的Who To Follow推荐服务,并能够将关注率提高15%。