Sloth: scaling laws for LLM skills to predict multi-benchmark performance across families
Department of Statistics, University of Michigan(密歇根大学统计学系);MIT-IBM Watson AI Lab, IBM Research(MIT-IBM Watson AI实验室,IBM研究);Computer Science and Artificial Intelligence Laboratory, MIT(MIT计算机科学与人工智能实验室);Institute of Foundation Models, MBZUAI(基础模型研究所,MBZUAI)
Scaling laws for large language models (LLMs) predict model performance based on parameters like size and training data. However, differences in training configurations and data processing across model families lead to significant variations in benchmark performance, making it difficult for a single scaling law to generalize across all LLMs. On the other hand, training family-specific scaling laws requires training models of varying sizes for every family. In this work, we propose Skills Scaling Laws (SSLaws, pronounced as Sloth), a novel scaling law that leverages publicly available benchmark data and assumes LLM performance is driven by low-dimensional latent skills, such as reasoning and instruction following. These latent skills are influenced by computational resources like model size and training tokens, but with varying efficiencies across model families. Sloth exploits correlations across benchmarks to provide more accurate and interpretable predictions while alleviating the need to train multiple LLMs per family. We present both theoretical results on parameter identification and empirical evaluations on 12 prominent benchmarks, from Open LLM Leaderboard v1/v2, demonstrating that Sloth predicts LLM performance accurately and offers insights into scaling behaviors for complex downstream tasks, increased test-time compute, and compute-optimal scaling of skills.
Automated discovery of symbolic laws governing skill acquisition from naturally occurring data
National Engineering Research Center of Educational Big Data, Central China Normal University(华中师范大学教育大数据国家工程研究中心);Faculty of Artificial Intelligence in Education, Central China Normal University(华中师范大学人工智能教育学部);National Engineering Research Center for E-learning, Central China Normal University(华中师范大学国家数字化学习工程技术研究中心)
A Tale of Tails: Model Collapse as a Change of Scaling Laws
Meta FAIR;Center for Data Science, New York University(纽约大学数据科学中心);School of Mathematical Sciences, Peking University(北京大学数学科学学院);Courant Institute, New York University(纽约大学柯朗研究所)
An exactly solvable model for emergence and scaling laws in the multitask sparse parity problem
Rudolf Peierls Centre for Theoretical Physics, University of Oxford(鲁道夫·皮尔士理论物理中心,牛津大学);Center for Quantum Structures in Modules and Spaces, Seoul National University(模块与空间中的量子结构中心,首尔国立大学)
SwiLTra-Bench: The Swiss Legal Translation Benchmark
ETH Zurich(苏黎世联邦理工学院);Swiss Federal Supreme Court(瑞士联邦最高法院);University of Zurich(苏黎世大学);University of Basel(巴塞尔大学);University of Geneva(日内瓦大学);University of Lausanne(洛桑大学);Canton of Solothurn(索洛图恩州);Max Planck Institute for Research on Collective Goods(马克斯·普朗克集体商品研究所)
Genius: A Generalizable and Purely Unsupervised Self-Training Framework For Advanced Reasoning
Shanghai AI Lab(上海人工智能实验室);Xi’an Jiaotong University(西安交通大学);The University of Hong Kong(香港大学);Peking University(北京大学);Hong Kong University of Science and Technology(香港科技大学)
TSRBench: A Comprehensive Multi-task Multi-modal Time Series Reasoning Benchmark for Generalist Models
Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);University of Illinois at Urbana–Champaign(伊利诺伊大学厄巴纳-香槟分校);University of Maryland, College Park(马里兰大学学院公园分校);University of California, San Diego(加州大学圣地亚哥分校)
Data Poisoning Vulnerabilities Across Healthcare AI Architectures: A Security Threat Analysis
Karolinska Institutet(卡罗林斯卡学院);KTH Royal Institute of Technology(KTH皇家理工学院);Karolinska University Hospital(卡罗林斯卡大学医院);University of Borås(布罗斯大学);Universidad Politécnica de Madrid(马德里理工大学)
A Latent Variable Framework for Scaling Laws in Large Language Models
Department of Statistics, University of Michigan(密歇根大学统计系);IBM Research and CSAIL, MIT(IBM研究与麻省理工学院计算机科学与人工智能实验室);Institute of Foundation Models, MBZUAI(MBZUAI基础模型研究所)