arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2606.03093cs.AI
收藏

分解提示如何引导行为

Decomposing how prompting steers behavior

Columbia University(哥伦比亚大学)

Fan L. Cheng, Nikolaus Kriegeskorte

AI总结:

提出嵌套几何分解框架,通过刺激不变映射分析提示如何重塑表示几何,揭示跨维度线性混合是提示引导行为的关键机制。

摘要:

提示引导大型语言模型(LLMs)和视觉语言模型(VLMs)无需权重更新,但指令变化如何重塑内部表示以产生行为仍不清楚。我们引入了一个嵌套几何分解框架,将提示视为对提示后内容表示几何的变换。对于每个提示对,我们使用越来越具表达力的刺激不变映射(平移、均匀缩放刚性变换、顺序轴缩放、仿射变换和非线性变换)对齐两个提示下相同刺激的表示。然后,我们通过将单个层的提示A隐藏状态替换为其映射版本,并测量提示B表示几何和行为的恢复程度,来因果测试每个映射。在三个LLM、三个VLM以及涵盖风格、情感、场景内容和数字的六个文本或图像数据集上,提示一致地将表示重塑为指示的任务结构。交叉验证的方差分解显示,许多提示诱导的激活变化由保持形状的映射(尤其是平移和均匀缩放刚性变换)捕获,而层级剖面揭示了跨层的模型和任务特定路由策略。关键的是,尽管平移和刚性层级已经改善了行为一致性,但仿射变换是第一个几乎完全恢复目标提示任务几何并带来相应行为增益的层级。这表明跨维度线性混合是提示将表示重组为指示任务结构的关键机制。我们的框架将提示诱导的表示变化分解为可解释的几何组件,并揭示了模型如何路由任务相关结构以产生提示驱动的行为。

英文摘要:

Prompting steers large language models (LLMs) and vision-language models (VLMs) without weight updates, but it remains unclear how instruction changes reshape internal representations to produce behavior. We introduce a nested geometric decomposition framework that treats prompting as a transformation of the representational geometry of the content following the prompt. For each prompt pair, we align representations of the same stimuli under two prompts using increasingly expressive stimulus-invariant maps: translation, rigid transformation with uniform scaling, sequential axis scaling, affine transformation, and nonlinear transformation. We then causally test each map by replacing a single layer's prompt-A hidden state for held-out stimuli with its mapped counterpart and measuring recovery of prompt-B representational geometry and behavior. Across three LLMs, three VLMs, and six text or image datasets spanning style, emotion, scene content, and number, prompts consistently reshape representations toward the instructed task structure. Cross-validated variance decomposition shows that much prompt-induced activation change is captured by shape-preserving maps, especially translation and rigid transformation with uniform scaling, while tier profiles reveal model- and task-specific routing strategies across layers. Crucially, although translation and rigid tiers already improve behavioral agreement, affine transformation is the first tier to nearly recover target-prompt task geometry and yields corresponding behavioral gains. This suggests that cross-dimensional linear mixing is a key mechanism by which prompts reorganize representations toward instructed task structure. Our framework decomposes prompt-induced representational change into interpretable geometric components and reveals how models route task-relevant structure to produce prompt-driven behavior.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑