Despite the superior performance, Large Language Models~(LLMs) require significant computational resources for deployment and use. To overcome this issue, quantization methods have been widely applied to reduce the memory footprint of LLMs as well as increasing the inference rate. However, a major challenge is that low-bit quantization methods often lead to performance degradation. It is important to understand how quantization impacts the capacity of LLMs. Different from previous studies focused on overall performance, this work aims to investigate the impact of quantization on \emph{emergent abilities}, which are important characteristics that distinguish LLMs from small language models. Specially, we examine the abilities of in-context learning, chain-of-thought reasoning, and instruction-following in quantized LLMs. Our empirical experiments show that these emergent abilities still exist in 4-bit quantization models, while 2-bit models encounter severe performance degradation on the test of these abilities. To improve the performance of low-bit models, we conduct two special experiments: (1) fine-gained impact analysis that studies which components (or substructures) are more sensitive to quantization, and (2) performance compensation through model fine-tuning. Our work derives a series of important findings to understand the impact of quantization on emergent abilities, and sheds lights on the possibilities of extremely low-bit quantization for LLMs.
Google Research（谷歌研究院）；Stanford University（斯坦福大学）；UNC Chapel Hill（北卡罗来纳大学教堂山分校）；DeepMind（深度思维）
Scaling up language models has been shown to predictably improve performance and sample efficiency on a wide range of downstream tasks. This paper instead discusses an unpredictable phenomenon that we refer to as emergent abilities of large language models. We consider an ability to be emergent if it is not present in smaller models but is present in larger models. Thus, emergent abilities cannot be predicted simply by extrapolating the performance of smaller models. The existence of such emergence implies that additional scaling could further expand the range of capabilities of language model…
Predictable Emergent Abilities of LLMs: Proxy Tasks Are All You Need
Beijing Academy of Artificial Intelligence (BAAI)（北京人工智能研究院（BAAI））；China University of Mining and Technology Beijing（中国矿业大学（北京））；Peking University（北京大学）
Predicting Emergent Abilities with Infinite Resolution Evaluation
Tsinghua University（清华大学）；Beijing Language and Culture University（北京语言大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Renmin University of China（中国人民大学）；Zhihu Inc.（知乎）；Modelbest Inc.（模贝斯特公司）
Arizona State University（亚利桑那州立大学）；Colorado State University（科罗拉多州立大学）
Intrigued by the claims of emergent reasoning capabilities in LLMs trained on general web corpora, in this paper, we set out to investigate their planning capabilities. We aim to evaluate (1) how good LLMs are by themselves in generating and validating simple plans in commonsense planning tasks (of the type that humans are generally quite good at) and (2) how good LLMs are in being a source of heuristic guidance for other agents--either AI planners or human planners--in their planning tasks. To investigate these questions in a systematic rather than anecdotal manner, we start by developing a b…
Saarland University（萨尔大学）；Microsoft Research India（微软印度研究院）
Scaling large language models (LLMs) leads to an emergent capacity to learn in-context from example demonstrations. Despite progress, theoretical understanding of this phenomenon remains limited. We argue that in-context learning relies on recombination of compositional operations found in natural language data. We derive an information-theoretic bound showing how in-context learning abilities arise from generic next-token prediction when the pretraining distribution has sufficient amounts of compositional structure, under linguistically motivated assumptions. A second bound provides a theoret…
AgentGroupChat: An Interactive Group Chat Simulacra For Better Eliciting Emergent Behavior
Fudan University（复旦大学）；Rhine AI（莱茵人工智能公司）；Shanghai University of Engineering and Science（上海工程技术大学）；Xiaohongshu Inc.（小红书公司）；Fudan-Aishu Cognitive Intelligence Joint Research Center（复旦-爱数认知智能联合研究中心）
University of Edinburgh（爱丁堡大学）；Allen Institute for AI（艾伦人工智能研究所）
The surprising ability of Large Language Models (LLMs) to perform well on complex reasoning with only few-shot chain-of-thought prompts is believed to emerge only in very large-scale models (100+ billion parameters). We show that such abilities can, in fact, be distilled down from GPT-3.5 ($\ge$ 175B) to T5 variants ($\le$ 11B). We propose model specialization, to specialize the model's ability towards a target task. The hypothesis is that large models (commonly viewed as larger than 100B) have strong modeling power, but are spread on a large spectrum of tasks. Small models (commonly viewed as…
The Impact of Quantization on Retrieval-Augmented Generation: An Analysis of Small LLMs
Amsterdam University of Applied Sciences（阿姆斯特丹应用科学大学）；Leiden University（莱顿大学）；Peoples' Friendship University of Russia (RUDN University)（俄罗斯人民友谊大学（鲁登大学））；Joint Institute for Nuclear Research（联合核子研究所）；Vrije Universiteit Amsterdam（阿姆斯特丹自由大学）；University of Skövde（舍夫德大学）
Brain-Inspired Two-Stage Approach: Enhancing Mathematical Reasoning by Imitating Human Thought Processes
School of Information Science and Technology, ShanghaiTech University（上海科技大学信息科学与技术学院）；Shanghai Innovation Center for Processor Technologies（上海处理器技术创新中心）；School of Information Science and Technology, University of Science and Technology of China（中国科学技术大学信息科学与技术学院）
An Empirical Study of Data Ability Boundary in LLMs' Math Reasoning
School of Information Science and Technology, ShanghaiTech University（上海科技大学信息科学与技术学院）；Shanghai Innovation Center for Processor Technologies（上海处理器技术创新中心）；School of Information Science and Technology, University of Science and Technology of China（中国科学技术大学信息科学与技术学院）
A Comprehensive Survey of Small Language Models in the Era of Large Language Models: Techniques, Enhancements, Applications, Collaboration with LLMs, and Trustworthiness
The Pennsylvania State University（宾夕法尼亚州立大学）；University of Pennsylvania（宾夕法尼亚大学）；UTHealth Houston（德克萨斯大学健康科学中心休斯顿分校）；Amazon（亚马逊公司）；Rensselaer Polytechnic Institute（伦斯勒理工学院）
The drivers of compositionality in artificial languages that emerge when two (or more) agents play a non-visual referential game has been previously investigated using approaches based on the REINFORCE algorithm and the (Neural) Iterated Learning Model. Following the more recent introduction of the \textit{Straight-Through Gumbel-Softmax} (ST-GS) approach, this paper investigates to what extent the drivers of compositionality identified so far in the field apply in the ST-GS context and to what extent do they translate into (emergent) systematic generalisation abilities, when playing a visual …
SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression
University of Washington（华盛顿大学）；HSE University（高等经济大学）；Yandex（扬德克斯）；Skoltech（斯科尔科沃科学技术研究院）；IST Austria（奥地利科学技术研究所）；ETH Zurich（苏黎世联邦理工学院）；NeuralMagic（神经魔法公司）
VQEL: Enabling Self-Play in Emergent Language Games via Agent-Internal Vector Quantization
Department of Computer Engineering（计算机工程系）；Sharif University of Technology（谢里夫理工学院）；School of Electrical and Computer Engineering（电气与计算机工程学院）；University of Tehran（德黑兰大学）
