arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2401.04757cs.LGcs.AI
收藏

语言模型基准测试性能的可预测性如何?

How predictable is language model benchmark performance?

Epoch

David Owen

AI总结:

研究11种模型架构在5个数量级计算扩展下的性能,发现平均基准测试性能可预测(外推1个数量级误差6pp),但单个任务预测误差较高(18pp),表明计算规模可作为预测AI能力的基础。

摘要:

我们研究了11种近期模型架构在跨越5个数量级的计算规模扩展下的大型语言模型性能。我们表明,正如在常用的BIG-Bench数据集中那样,聚合众多单独任务和评估的平均基准测试性能,作为训练计算规模的函数具有相当的可预测性。具体而言,在跨越1个数量级的计算量外推BIG-Bench Hard性能时,我们观察到平均绝对误差为6个百分点(pp)。相比之下,在跨越1个数量级的计算量对单个BIG-Bench任务进行外推时,会产生更高的18pp平均误差。尽管如此,单个任务的性能仍然显著比随机猜测更具可预测性。总体而言,我们的工作表明,计算规模扩展为预测多样化基准测试中的AI能力提供了有希望的基础,尽管预测特定任务的性能仍面临挑战。

英文摘要:

We investigate large language model performance across five orders of magnitude of compute scaling in eleven recent model architectures. We show that average benchmark performance, aggregating over many individual tasks and evaluations as in the commonly-used BIG-Bench dataset, is decently predictable as a function of training compute scale. Specifically, when extrapolating BIG-Bench Hard performance across one order of magnitude in compute, we observe average absolute errors of 6 percentage points (pp). By contrast, extrapolation for individual BIG-Bench tasks across an order of magnitude in compute yields higher average errors of 18pp. Nonetheless, individual task performance remains significantly more predictable than chance. Overall, our work suggests compute scaling provides a promising basis to forecast AI capabilities in diverse benchmarks, though predicting performance in specific tasks poses challenges.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑