State-of-the-art LLMs are powered by scaling -- scaling model size, dataset size and cluster size. It is economically infeasible to extensively tune hyperparameter for the largest runs. Instead, approximately optimal hyperparameters must be inferred or \textit{transferred} from smaller experiments. Hyperparameter transfer across model sizes has been studied in Yang et al. However, hyperparameter transfer across dataset size -- or token horizon -- has not been studied yet. To remedy this we conduct a large scale empirical study on how optimal learning rate (LR) depends on token horizon in LLM training. We first demonstrate that the optimal LR changes significantly with token horizon -- longer training necessitates smaller LR. Secondly we demonstrate the the optimal LR follows a scaling law, and that the optimal LR for longer horizons can be accurately estimated from shorter horizons via such scaling laws. We also provide a rule-of-thumb for transferring LR across token horizons with zero overhead over current practices. Lastly we provide evidence that LLama-1 used too high LR, and estimate the performance hit from this. We thus argue that hyperparameter transfer across data size is an important and overlooked component of LLM training.
Scaling Capability in Token Space: An Analysis of Large Vision Language Model
School of Automation, Guangdong University of Technology(广东工业大学自动化学院);Key Laboratory of Intelligent Detection and the Internet of Things in Manufacturing, Ministry of Education(教育部智能制造智能检测与物联网重点实验室);Guangdong Provincial Key Laboratory of Intelligent Systems and Optimization Integration(广东省智能系统与优化集成重点实验室);Medical Science Data-driven Mathematics Team, RIKEN Center for Interdisciplinary Theoretical and Mathematical Sciences(RIKEN跨学科理论与数学科学中心医学科学数据驱动数学团队);Medical Data Mathematical Reasoning Special Team, RIKEN Center for Integrative Medical Sciences(RIKEN整合医学科学中心医学数据数学推理特别团队);Department of Artificial Intelligence Medicine, Chiba University(千叶大学人工智能医学系);Tensor Learning Team, RIKEN Center for Advanced Intelligence Project(RIKEN高级人工智能项目中心张量学习团队)
The Optimal Token Baseline: Variance Reduction for Long-Horizon LLM-RL
National University of Singapore(新加坡国立大学);Nanyang Technological University(南洋理工大学);Fudan University(复旦大学);Hong Kong University of Science(香港科学大学);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Vector Institute(向量研究所)
Scaling-Aware Adapter for Structure-Grounded LLM Reasoning
Department of Computer Science, Western University, London, Canada(加拿大伦敦西方大学计算机科学系);Department of Biochemistry, Western University, London, Canada(加拿大伦敦西方大学生物化学系)
How Many Visual Tokens Do Multimodal Language Models Need? Scaling Visual Token Pruning with F^3A
School of Computer Science and Engineering, Northeastern University, Shenyang 110819, China(东北大学计算机科学与工程学院,沈阳 110819,中国);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);School of Computer and Communication Engineering, Northeastern University, Qinhuangdao 066004, China(东北大学计算机与通信工程学院,秦皇岛 066004,中国)
Token-Disentangled Latent Test-Time Scaling for Vision-Language Reasoning
Qwen Business Unit of Alibaba(阿里巴巴通义千问事业部);School of Artificial Intelligence, Nanjing University(南京大学人工智能学院);National Key Laboratory for Novel Software Technology, Nanjing University(南京大学计算机软件新技术全国重点实验室);Zhejiang University(浙江大学);University of Waterloo(滑铁卢大学);Chinese Academy of Sciences(中国科学院);The Hong Kong University of Science and Technology(香港科技大学);Frontier Robotics(前沿机器人)
Critical attention scaling in long-context transformers
Department of Mathematics, Massachusetts Institute of Technology(数学系,麻省理工学院);Department of Electrical Engineering and Computer Science, Massachusetts Institute of Technology(电气工程与计算机科学系,麻省理工学院)
Efficient and Trainable Language Model Test-Time Scaling via Local Branch Routing
Northwestern University(西北大学);Rutgers University(罗格斯大学);University of Wisconsin–Madison(威斯康星大学麦迪逊分校);Carnegie Mellon University(卡内基梅隆大学);LMSYS Org(LMSYS组织);Tilde Research(Tilde研究);University of California, Santa Barbara(加州大学圣塔芭芭拉分校);University of Toronto(多伦多大学);University of British Columbia(不列颠哥伦比亚大学);University of California, San Diego(加州大学圣迭戈分校)
OmniMouse: Scaling properties of multi-modal, multi-task Brain Models on 150B Neural Tokens
Department of Ophthalmology, Byers Eye Institute, Stanford University(斯坦福大学眼科学系、比尔斯眼科研究所);Stanford Bio-X, Stanford University(斯坦福大学生物交叉学科);Wu Tsai Neurosciences Institute, Stanford University(斯坦福大学吴泰教授神经科学研究所);Institute of Computer Science and Campus Institute Data Science, University Göttingen(哥廷根大学计算机科学研究所和校园数据科学研究所)
Learning a Continue-Thinking Token for Enhanced Test-Time Scaling
Department of Computer Science, Technion – Israel Institute of Technology(计算机科学系,技术学院–以色列理工学院);Department of Electrical and Computer Engineering, Technion – Israel Institute of Technology(电气与计算机工程系,技术学院–以色列理工学院)