Prompting steers large language models (LLMs) and vision-language models (VLMs) without weight updates, but it remains unclear how instruction changes reshape internal representations to produce behavior. We introduce a nested geometric decomposition framework that treats prompting as a transformation of the representational geometry of the content following the prompt. For each prompt pair, we align representations of the same stimuli under two prompts using increasingly expressive stimulus-invariant maps: translation, rigid transformation with uniform scaling, sequential axis scaling, affine transformation, and nonlinear transformation. We then causally test each map by replacing a single layer's prompt-A hidden state for held-out stimuli with its mapped counterpart and measuring recovery of prompt-B representational geometry and behavior. Across three LLMs, three VLMs, and six text or image datasets spanning style, emotion, scene content, and number, prompts consistently reshape representations toward the instructed task structure. Cross-validated variance decomposition shows that much prompt-induced activation change is captured by shape-preserving maps, especially translation and rigid transformation with uniform scaling, while tier profiles reveal model- and task-specific routing strategies across layers. Crucially, although translation and rigid tiers already improve behavioral agreement, affine transformation is the first tier to nearly recover target-prompt task geometry and yields corresponding behavioral gains. This suggests that cross-dimensional linear mixing is a key mechanism by which prompts reorganize representations toward instructed task structure. Our framework decomposes prompt-induced representational change into interpretable geometric components and reveals how models route task-relevant structure to produce prompt-driven behavior.
DR.GAP: Mitigating Bias in Large Language Models using Gender-Aware Prompting with Decoupled Reasoning
School of Information Science and Technology, ShanghaiTech University(上海科技大学信息科学与技术学院);School of Computer and Cyber Sciences, Augusta University(奥古斯塔大学计算机与网络科学学院)
Toward universal steering and monitoring of AI models
Computer Science and Engineering(计算机科学与工程);Broad Institute of MIT and Harvard(MIT和哈佛大学Broad研究所);UC San Diego(圣地亚哥大学);Harvard SEAS(哈佛大学工程与应用科学学院);MIT Mathematics(MIT数学系);Halıcıoğlu Data Science Institute(Halıcıoğlu数据科学研究所);Harvard CMSA(哈佛大学计算机科学与应用数学系)
Explaining Intrinsic Moral Self-Correction with Mechanistic Interpretability
Graduate Institute of Communication Engineering, National Taiwan University, Taipei, Taiwan(通讯工程研究院,国立台湾大学);MediaTek Research, Taipei, Taiwan(联发科研究,台北,台湾);Department of Electrical Engineering, National Taiwan University, Taipei, Taiwan(电气工程系,国立台湾大学);Department of Electrical Engineering, National Tsing Hua University, Hsinchu, Taiwan(电气工程系,国立清华大学);AI Research Center (AINTU), National Taiwan University, Taipei, Taiwan(人工智能研究中心(AINTU),国立台湾大学)
HiFo-Prompt: Prompting with Hindsight and Foresight for LLM-based Automatic Heuristic Design
School of Mathematics and Statistics, Xi’an Jiaotong University, Xi’an, China(西安交通大学数学与统计学院);School of Electronics and Information, Northwest Polytechnical University, Xi’an, China(西北工业大学电子与信息学院)