Large language models like GPT-4 exhibit emergent capabilities across general-purpose tasks, such as basic arithmetic, when trained on extensive text data, even though these tasks are not explicitly encoded by the unsupervised, next-token prediction objective. This study investigates how small transformers, trained from random initialization, can efficiently learn arithmetic operations such as addition, multiplication, and elementary functions like square root, using the next-token prediction objective. We first demonstrate that conventional training data is not the most effective for arithmetic learning, and simple formatting changes can significantly improve accuracy. This leads to sharp phase transitions as a function of training data scale, which, in some cases, can be explained through connections to low-rank matrix completion. Building on prior work, we then train on chain-of-thought style data that includes intermediate step results. Even in the complete absence of pretraining, this approach significantly and simultaneously improves accuracy, sample complexity, and convergence speed. We also study the interplay between arithmetic and text data during training and examine the effects of few-shot prompting, pretraining, and model scale. Additionally, we discuss length generalization challenges. Our work highlights the importance of high-quality, instructive data that considers the particular characteristics of the next-word prediction objective for rapidly eliciting arithmetic capabilities.
Provable In-Context Vector Arithmetic via Retrieving Task Concepts
Department of Computer Science, City University of Hong Kong, Hong Kong SAR；Center for Advanced Intelligence Project, RIKEN, Japan；Institute of High Performance Computing (IHPC), Agency for Science, Technology；Research (A STAR), Singapore；Centre for Frontier AI Research (CFAR), Agency for Science, Technology；College of Computing；Data Science, Nanyang Technological University, Singapore；Department of Mathematical Informatics, The University of Tokyo, Japan；School of Mathematics；Statistics, The University of Sydney, Australia
Teaching Transformers to Solve Combinatorial Problems through Efficient Trial & Error
National Technical University of Athens（希腊雅典国家技术大学）；National and Kapodistrian University of Athens（希腊雅典国家与卡波迪斯特里亚大学）；Archimedes, Athena Research Center（阿提卡研究中心阿基米德）
LIB-KD: Teaching Inductive Bias for Efficient Vision Transformer Distillation and Compression
Department of Electrical Engineering Indian Institute of Technology Delhi（电子工程系印度理工学院德里）；Yogananda School of AI, Computers and Data Sciences（Yoganananda人工智能、计算机与数据科学学院）
Princeton University（普林斯顿大学）；EPFL（洛桑联邦理工学院）；New York University（纽约大学）；Stanford University（斯坦福大学）；Google Research（谷歌研究院）；Carnegie Mellon University（卡内基梅隆大学）；Microsoft Research（微软研究院）；Meta AI
Transformers for Learning on Noisy and Task-Level Manifolds: Approximation and Generalization Insights
School of Mathematics, Georgia Institute of Technology（佐治亚理工学院数学系）；Department of Mathematics, Purdue University（普渡大学数学系）；Department of Mathematics and Halicioğlu Data Science Institute, University of California, San Diego（加州大学圣地亚哥分校数学系和Halicioğlu数据科学研究所）
Shattered Compositionality: Counterintuitive Learning Dynamics of Transformers for Arithmetic
University of Wisconsin–Madison（威斯康星大学麦迪逊分校）；Department of Computer Sciences, University of Wisconsin–Madison（威斯康星大学麦迪逊分校计算机科学系）；Department of Statistics, University of Wisconsin–Madison（威斯康星大学麦迪逊分校统计学系）
Transformers Can Do Arithmetic with the Right Embeddings
University of Maryland（马里兰大学）；Lawrence Livermore National Laboratory（劳伦斯利弗莫尔国家实验室）；ELLIS Institute Tübingen（图宾根ELLIS研究所）；Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；Carnegie Mellon University（卡内基梅隆大学）
Fourier Circuits in Neural Networks and Transformers: A Case Study of Modular Arithmetic with Multiple Inputs
Fuzhou University（福州大学）；The University of Hong Kong（香港大学）；University of Wisconsin-Madison（威斯康星大学麦迪逊分校）；The Simons Institute for the Theory of Computing at the University of California, Berkeley（加州大学伯克利分校西蒙斯理论计算研究所）；University of Southern California（南加州大学）
How Numerical Precision Affects Arithmetical Reasoning Capabilities of LLMs
State Key Laboratory of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University（通用人工智能国家重点实验室，智能科学与技术学院，北京大学）；School of EECS, Peking University（电子工程系，北京大学）；Huawei Noah’s Ark Lab（华为诺亚实验室）；Center for Machine Learning Research, Peking University（机器学习研究中心，北京大学）
