Scaling large language models (LLMs) leads to an emergent capacity to learn in-context from example demonstrations. Despite progress, theoretical understanding of this phenomenon remains limited. We argue that in-context learning relies on recombination of compositional operations found in natural language data. We derive an information-theoretic bound showing how in-context learning abilities arise from generic next-token prediction when the pretraining distribution has sufficient amounts of compositional structure, under linguistically motivated assumptions. A second bound provides a theoretical justification for the empirical success of prompting LLMs to output intermediate steps towards an answer. To validate theoretical predictions, we introduce a controlled setup for inducing in-context learning; unlike previous approaches, it accounts for the compositional nature of language. Trained transformers can perform in-context learning for a range of tasks, in a manner consistent with the theoretical results. Mirroring real-world LLMs in a miniature setup, in-context learning emerges when scaling parameters and data, and models perform better when prompted to output intermediate steps. Probing shows that in-context learning is supported by a representation of the input's compositional structure. Taken together, these results provide a step towards theoretical understanding of emergent behavior in large language models.
Legal Rule Induction: Towards Generalizable Principle Discovery from Analogous Judicial Precedents
Department of Computer Science and Engineering, HKUST, Hong Kong SAR, China（香港科技大学计算机科学与工程系）；School of Law, Tsinghua University, Beijing, China（清华大学法学院）
Learning to Evaluate Before Improving: Automatic Rubric Induction for Automatic Research Agents
Zhejiang University（浙江大学）；University of Electronic Science and Technology of China（电子科技大学）；Beijing University of Posts and Telecommunications（北京邮电大学）；Zhejiang University of Technology（浙江工业大学）
Generative Emergent Communication: Large Language Model is a Collective World Model
Graduate School of Informatics, Kyoto University（京都大学信息科学研究生院）；Graduate School of Engineering, The University of Tokyo（东京大学工学研究生院）；Graduate School of Informatics and Engineering, The University of Electro-Communications（电通大学信息科学和工学研究生院）；College of Information Science and Engineering, Ritsumeikan University（立命馆大学信息科学和工学学院）
University of Amsterdam（阿姆斯特丹大学）；Institute of Cognitive Science（认知科学研究所）；University of Osnabruek（奥斯纳布吕克大学）；Institute for Logic, Language and Computation（逻辑、语言与计算研究所）
