arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2605.16591cs.LGcs.AI
收藏

少样本示例如何累加:上下文学习中函数向量的因果分解

How Few-Shot Examples Add Up: A Causal Decomposition of Function Vectors in In-Context Learning

Entang Wang, Yiwei Wang, Aleksandra Bakalova, Michael Hahn

AI总结:

本文通过因果分解揭示少样本提示中函数向量由示例级子向量线性组合而成,并发现模型通过注意力重加权机制根据上下文调整示例贡献。

摘要:

上下文学习(ICL)擅长从极少量示例中学习新任务,但我们仍缺乏对少样本提示如何塑造模型函数向量(FV)——一种驱动ICL查询任务行为的因果激活方向——的机制性解释。跨任务和模型,一个$n$样本FV可以通过示例级子FV的线性组合很好地近似,表明来自单个演示的贡献具有加性和可组合性。除了加性之外,我们展示了模型基于先前示例对单个示例的表示进行上下文化,以自适应地重新加权哪些演示主导FV:注意力转向在上下文中信息量更大、歧义更少的示例。最后,因果分解将查询-键路由与值更新分离,发现上下文化对FV质量最一致的贡献来自查询-键对齐——尤其是在歧义设置中——而值介导的效应则更加异质。综合起来,这些结果将加性叠加与上下文相关的注意力重加权统一为一个机制性的、可检验的说明,解释少样本提示如何实现任务。

英文摘要:

In-context learning (ICL) excels at new tasks from minimal examples, yet we still lack a mechanistic explanation of how few-shot prompts shape a model's function vector (FV)--a causal activation direction that drives task behavior on the ICL query. Across tasks and models, an $n$-shot FV is well-approximated by a linear combination of example-level sub-FVs, suggesting additive and composable contributions from individual demonstrations. Beyond additivity, we show that models contextualize individual examples' representations based on prior examples to adaptively reweight which demonstrations dominate the FV: attention shifts toward examples that are more informative and less ambiguous under the context. Finally, a causal decomposition separates Query-Key routing from Value updates, finding that contextualization's most consistent contributions to FV quality arise from Query-Key alignment--particularly in ambiguous settings--while Value-mediated effects are more heterogeneous. Together, these results unify additive superposition with context-dependent attention reweighting into a mechanistic, testable account of how few-shot prompts implement tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑