While induction is considered a key mechanism for in-context learning in LLMs, understanding its precise circuit decomposition beyond toy models remains elusive. Here, we study the emergence of induction behavior within LLMs by probing their response to weak single-token perturbations of the residual stream. We find that LLMs exhibit a robust, universal regime in which their response remains scale-invariant under changes in perturbation strength, thereby allowing us to quantify the build-up of token correlations throughout the model. By applying our method, we observe signatures of induction behavior within the residual stream of Gemma-2-2B, Llama-3.2-3B, and GPT-2-XL. Across all models, we find that these induction signatures gradually emerge within intermediate layers and identify the relevant model sections composing this behavior. Our results provide insights into the collective interplay of components within LLMs and serve as a benchmark for large-scale circuit analysis.
ItD: Large Language Models Can Teach Themselves Induction through Deduction
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);University of Chinese Academy of Sciences(中国科学院大学);Xiaohongshu Inc(小红书公司);Texas A&M University(德克萨斯农工大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室)
Temporal Knowledge Question Answering via Abstract Reasoning Induction
Laboratory for Big Data and Decision, National University of Defense Technology(国防科技大学大数据与决策实验室);Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳))
Neuron-based Personality Trait Induction in Large Language Models
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学首都人工智能学院);Tongyi Lab(通义实验室);Institute of Statistics and Big Data, Renmin University of China(中国人民大学统计与大数据研究院)
Identifying Semantic Induction Heads to Understand In-Context Learning
Shanghai Jiao Tong University(上海交通大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Fudan University(复旦大学);The Chinese University of Hong Kong(香港中文大学)
Strategy-Induct: Task-Level Strategy Induction for Instruction Generation
Department of Computer Science and Information Engineering, National Taiwan University, Taiwan(国立台湾大学计算机科学与资讯工程学系);Institute of Information Science, Academia Sinica, Taiwan(台湾“中央研究院”资讯科学研究所);AI Research Center (AINTU), National Taiwan University, Taiwan(国立台湾大学人工智能研究中心)
CHAIRO: Contextual Hierarchical Analogical Induction and Reasoning Optimization for LLMs
National Engineering Laboratory for Big Data System Computing Technology(国家大数据系统计算技术工程实验室);Shenzhen University(深圳大学);Tsinghua University(清华大学);National University of Singapore(新加坡国立大学)
The Staircase of Ethics: Probing LLM Value Priorities through Multi-Step Induction to Complex Moral Dilemmas
University of Chinese Academy of Sciences(中国科学院大学);Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);Media Synthesis and Forensics Lab(媒体合成与取证实验室);Zhongguancun Laboratory(中关村实验室)
Resource-Lean Lexicon Induction for German Dialects
MaiNLP, Center for Information and Language Processing, LMU Munich, Germany(MaiNLP,信息与语言处理中心,慕尼黑大学,德国);Munich Center for Machine Learning (MCML), Munich, Germany(慕尼黑机器学习中心(MCML),慕尼黑,德国)
Legal Rule Induction: Towards Generalizable Principle Discovery from Analogous Judicial Precedents
Department of Computer Science and Engineering, HKUST, Hong Kong SAR, China(香港科技大学计算机科学与工程系);School of Law, Tsinghua University, Beijing, China(清华大学法学院)
Stream Aligner: Efficient Sentence-Level Alignment via Distribution Induction
Institute for AI, Peking University(北京大学人工智能研究院);State Key Laboratory of General Artificial Intelligence, Institute for AI, Peking University(北京大学人工智能研究院通用人工智能国家重点实验室)
Diverge to Induce Prompting: Multi-Rationale Induction for Zero-Shot Reasoning
Department of Computer Science and Information Engineering, National Taiwan University, Taiwan(国家台湾大学计算机科学与信息工程系);Institute of Information Science, Academia Sinica, Taiwan(学术院信息科学研究所);AI Research Center (AINTU), National Taiwan University, Taiwan(国家台湾大学人工智能研究中心(AINTU))
Emergence and Localisation of Semantic Role Circuits in LLMs
Department of Computer Science, University of Manchester(曼彻斯特大学计算机科学系);Idiap Research Institute(Idiap研究 institute);National Biomarker Centre, CRUK-MI, Univ. of Manchester(国家生物标志物中心,CRUK-MI,曼彻斯特大学)
The Emergence of Abstract Thought in Large Language Models Beyond Any Language
National University of Singapore(新加坡国立大学);Salesforce AI Research(Salesforce AI研究院);Peking University(北京大学);Singapore University of Technology and Design(新加坡科技设计大学)
DHI: Leveraging Diverse Hallucination Induction for Enhanced Contrastive Factuality Control in Large Language Models
School of Computer Science, Wuhan University(武汉大学计算机学院);Tsinghua Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院);School of Artificial Intelligence, Wuhan University(武汉大学人工智能学院)