arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2412.15519cs.LG
收藏

PreNeT:利用计算特征预测深度神经网络训练时间

PreNeT: Leveraging Computational Features to Predict Deep Neural Network Training Time

York University(约克大学)

Alireza Pourali, Arian Boukani, Hamzeh Khazaei

AI总结:

PreNeT通过整合逐层参数、算术运算和内存利用等计算特征,预测深度神经网络在不同硬件上的训练时间,实验显示其预测准确率比现有最先进框架提升高达72%。

摘要:

训练深度学习模型,尤其是基于Transformer的架构(如大型语言模型LLMs),需要大量的计算资源和较长的训练周期。虽然优化配置和基础设施选择可以显著降低相关成本,但这种优化需要借助初步分析工具。本文介绍了PreNeT,一种旨在应对这一优化挑战的新型预测框架。PreNeT通过整合全面的计算指标来促进训练优化,这些指标包括逐层参数、算术运算和内存利用情况。PreNeT的一个关键特性是,它能够准确预测在先前未检验过的硬件基础设施(包括新型加速器架构)上的训练时长。该框架采用一种精细的方法来捕捉和分析各种神经网络层的独特特征,从而增强现有的预测方法。通过主动部署PreNeT,研究人员和从业者可以确定最优配置、参数设置和硬件规格,以最大限度地提高成本效率并缩短训练时间。实验结果表明,与当前最先进的框架相比,PreNeT在预测准确率上实现了高达72%的提升。

英文摘要:

Training deep learning models, particularly Transformer-based architectures such as Large Language Models (LLMs), demands substantial computational resources and extended training periods. While optimal configuration and infrastructure selection can significantly reduce associated costs, this optimization requires preliminary analysis tools. This paper introduces PreNeT, a novel predictive framework designed to address this optimization challenge. PreNeT facilitates training optimization by integrating comprehensive computational metrics, including layer-specific parameters, arithmetic operations and memory utilization. A key feature of PreNeT is its capacity to accurately predict training duration on previously unexamined hardware infrastructures, including novel accelerator architectures. This framework employs a sophisticated approach to capture and analyze the distinct characteristics of various neural network layers, thereby enhancing existing prediction methodologies. Through proactive implementation of PreNeT, researchers and practitioners can determine optimal configurations, parameter settings, and hardware specifications to maximize cost-efficiency and minimize training duration. Experimental results demonstrate that PreNeT achieves up to 72% improvement in prediction accuracy compared to contemporary state-of-the-art frameworks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑