arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2601.06649cs.LGcs.AI
收藏

重新审视训练规模:关于令牌计数、功耗和参数效率的实证研究

Revisiting Training Scale: An Empirical Study of Token Count, Power Consumption, and Parameter Efficiency

ECPI University(ECPI大学)

Joe Dwyer

AI总结:

通过固定硬件和训练条件的重复测量实验,发现增加训练令牌数会导致训练效率严格单调下降,即使性能有边际提升,也表明能耗效率低下。

摘要:

机器学习研究质疑了训练令牌数的增加是否能在大型语言模型中可靠地产生比例性能提升。基于先前引入能量感知参数效率度量的工作,本研究实证检验了在固定硬件和训练条件下增加训练令牌数的影响。本工作的重要性在于将功耗和执行时长(如功率采样频率所反映的)明确整合到令牌规模分析中,这解决了先前研究强调性能结果而低估计算和能量成本的空白。通过在恒定GPU实例上使用相同模型架构、优化器设置和轮次数的重复测量实验设计,训练了一个11亿参数的TinyLlama模型,使用三个令牌数(500K、1M和2M)。虽然传统性能指标在令牌规模上表现出不一致或递减的回报,但包含功耗和执行时长后,揭示了随着令牌数增加,训练效率严格单调下降。重复测量方差分析表明令牌数对参数效率有强效应,所有配对比较在Bonferroni校正后仍然显著。这些发现表明,即使观察到边际性能提升,增加训练令牌数可能在能量上效率低下,强调了在大型语言模型训练中效率感知评估的重要性。

英文摘要:

Research in machine learning has questioned whether increases in training token counts reliably produce proportional performance gains in large language models. Building on prior work introducing an energy-aware parameter efficiency metric, this study empirically examines the effects of increasing training token counts under fixed hardware and training conditions. The significance of this work lies in the explicit integration of power consumption and execution duration, as reflected by the power sampling frequency, into token-scale analysis. This addresses a gap in prior studies emphasizing performance outcomes while underrepresenting computational and energy costs. Using a repeated-measures experimental design on a constant GPU instance with an identical model architecture, optimizer settings, and epoch counts, a 1.1-billion-parameter TinyLlama model was trained at three token counts (500K, 1M, and 2M). While conventional performance metrics exhibited inconsistent or diminishing returns across token scales, the inclusion of power consumption and execution duration revealed a strictly monotonic decline in training efficiency as token count increased. Repeated-measures ANOVA demonstrated a strong effect of token count on parameter efficiency, with all pairwise comparisons remaining significant following Bonferroni correction. These findings indicate that increases in training token counts may be energetically inefficient even when marginal performance improvements are observed, underscoring the importance of efficiency-aware evaluation in large language model training.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑