arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2412.04403cs.CLcs.AI
收藏

通过计算高效模型阶梯建立任务缩放定律

Establishing Task Scaling Laws via Compute-Efficient Model Ladders

Allen Institute for AI(艾伦人工智能研究所);University of Washington(华盛顿大学);Princeton University(普林斯顿大学)
Akshita Bhagia, Jiacheng Liu, Alexander Wettig, David Heineman, Oyvind Tafjord, Ananya Harsh Jha, Luca Soldaini, Noah A. Smith, Dirk Groeneveld, Pang Wei Koh, J… 展开作者

Akshita Bhagia, Jiacheng Liu, Alexander Wettig, David Heineman, Oyvind Tafjord, Ananya Harsh Jha, Luca Soldaini, Noah A. Smith, Dirk Groeneveld, Pang Wei Koh, Jesse Dodge, Hannaneh Hajishirzi

AI总结:

本文提出通过训练小规模阶梯模型建立任务缩放定律,以极低计算成本预测大规模预训练语言模型在具体任务上的表现,并在多个任务上实现了高精度预测。

摘要:

我们开发了任务缩放定律和模型阶梯,用于在过度训练设置下预测预训练语言模型(LMs)在单个任务上的表现。标准的语言建模损失幂律无法准确建模任务表现。因此,我们采用两步预测方法:(1)使用模型规模和数据规模预测中间损失,然后(2)利用该中间损失预测任务表现。我们训练一组小规模“阶梯”模型,收集数据点以拟合两个预测步骤的参数化函数,并对两个目标模型进行预测:一个训练至4T token的7B模型和一个训练至5T token的13B模型。训练阶梯模型仅花费目标模型所用计算量的1%。在四个以排序分类形式构建的多项选择题任务上,我们能够以2个点的绝对误差预测两个目标模型的准确率。我们发现,预测误差较高的任务在模型检查点上的指标方差也更高。我们还对比了多种预测准确率的设计选择,并提出了将我们的方法扩展到新模型和新任务的建议。

英文摘要:

We develop task scaling laws and model ladders to predict the individual task performance of pretrained language models (LMs) in the overtrained setting. Standard power laws for language modeling loss cannot accurately model task performance. Therefore, we leverage a two-step prediction approach: (1) use model and data size to predict an intermediate loss, then (2) use it to predict task performance. We train a set of small-scale "ladder" models, collect data points to fit the parameterized functions of the two prediction steps, and make predictions for two target models: a 7B model trained to 4T tokens and a 13B model trained to 5T tokens. Training the ladder models only costs 1% of the compute used for the target models. On four multiple-choice tasks formatted as ranked classification, we can predict the accuracy of both target models within 2 points of absolute error. We find that tasks with higher prediction error also have higher variance in the metrics over model checkpoints. We also contrast multiple design choices for predicting accuracy, and present recommendations for extending our method to new models and tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑