arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2604.01025cs.LGcs.AI
收藏

快速且准确地探测训练中的LLM下游性能

Fast and Accurate Probing of In-Training LLMs' Downstream Performances

Southern University of Science and Technology(南方科技大学);Huawei(华为)

Zhichen Liu, Tianle Lun, Yang Xu, Zhibin Wen, Hao An, Yulin Ou, Jianhui Xu, Hao Zhang, Wenyi Fang, Yang Zheng

AI总结:

本文提出一种高效准确的训练中LLM下游性能评估方法,通过轻量级探针预测模型在下游任务上的表现,减少计算延迟并提升评估效率。

摘要:

在参数规模和测试时间上扩展大型语言模型(LLMs)的范式推动了AI能力的边界,但使传统的生成评估范式变得成本过高,从而使得LLM训练中的下游性能评估延迟不可接受。然而,简单的指标如训练损失(困惑度)并不总与下游性能相关,因为有时它们的趋势与实际任务结果 diverge。为了解决这一难题,我们引入了一种新的训练中评估范式,利用轻量级探针监控下游性能。探针以LLM检查点(在训练期间)的内部表示作为输入,并直接预测检查点在下游任务上的表现(以成功率概率(即pass@1)衡量)。我们设计了几种探针架构,并通过OLMo3-7B的检查点在多样化的下游任务上验证了其有效性。探针可以准确预测检查点的表现(平均AUROC>0.75),具有良好的泛化能力(早期预测后期),并将计算延迟从约1小时(使用传统生成评估方法)减少到约3分钟。总之,本文提出了一种实用且可扩展的训练中下游评估范式,使LLM开发过程更加敏捷、有信息且高效。

英文摘要:

The paradigm of scaling Large Language Models (LLMs) in both parameter size and test time has pushed the boundaries of AI capabilities, but at the cost of making the traditional generative evaluation paradigm prohibitively expensive, therefore making the latency of LLM's in-training downstream performance evaluation unbearable. However, simple metrics like training loss (perplexity) are not always correlated with downstream performance, as sometimes their trends diverge from the actual task outcomes. This dilemma calls for a method that is computationally efficient and sufficiently accurate in measuring model capabilities. To address this challenge, we introduce a new in-training evaluation paradigm that uses a lightweight probe for monitoring downstream performance. The probes take the internal representations of LLM checkpoints (during training) as input and directly predict the checkpoint's performance on downstream tasks measured by success probability (i.e., pass@1). We design several probe architectures, validating their effectiveness using the OLMo3-7B's, OLMo3-32B's, and Apertus-8B's checkpoints across a diverse set of downstream tasks. The probes can accurately predict a checkpoint's performance (with avg. AUROC$>$0.75), have decent generalizability across checkpoints (earlier predicts later), and reduce the computation latency from $\sim$1 hr (using conventional generative evaluation method) to $\sim$3 min. In sum, this work presents a practical and scalable in-training downstream evaluation paradigm, enabling a more agile, informed, and efficient LLM development process.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑