arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.19611cs.LG
收藏

演示、推理链和提示:对基于上下文学习(ICL)的理论分析

Demonstrations, CoT, and Prompting: A Theoretical Analysis of ICL

Xuhan Tong, Yuchen Zeng, Jiawei Zhang

AI总结:

本文通过理论分析探讨了基于上下文学习(ICL)中演示、推理链提示和提示模板的影响,揭示了模型泛化能力的决定因素。

摘要:

基于上下文学习(ICL)使预训练大语言模型能够通过条件于少量输入输出演示来适应下游任务,而无需任何参数更新。尽管已有许多理论研究解释ICL的工作原理,但大多数研究要么依赖于强架构或数据假设,要么未能捕捉到关键实际因素如演示选择、推理链(CoT)提示、演示数量和提示模板的影响。本文通过在温和假设下建立ICL的理论分析,将这些设计选择与泛化行为联系起来。我们推导了ICL测试损失的上界,表明性能由(i)所选演示的质量(通过连接测试提示到预训练样本路径上的ICL损失的Lipschitz常数量化)、(ii)预训练模型的内在ICL能力,以及(iii)分布偏移程度决定。在同一框架下,我们分析了推理链提示作为任务分解的诱导作用,并表明当每个子步骤中演示选择良好且子任务更易学习时,它是有益的。最后,我们描述了ICL对提示模板的敏感性如何随着演示数量的变化而变化。总的来说,本文研究表明预训练使模型能够超越观察到的任务进行泛化,而推理链使模型能够将更简单的子任务组合成更复杂的任务,演示和指令使模型能够检索相似或复杂的任务,包括可以组合成更复杂任务的任务,共同支持泛化到未见任务。所有理论见解都通过实验得到验证。

英文摘要:

In-Context Learning (ICL) enables pretrained LLMs to adapt to downstream tasks by conditioning on a small set of input-output demonstrations, without any parameter updates. Although there have been many theoretical efforts to explain how ICL works, most either rely on strong architectural or data assumptions, or fail to capture the impact of key practical factors such as demonstration selection, Chain-of-Thought (CoT) prompting, the number of demonstrations, and prompt templates. We address this gap by establishing a theoretical analysis of ICL under mild assumptions that links these design choices to generalization behavior. We derive an upper bound on the ICL test loss, showing that performance is governed by (i) the quality of selected demonstrations, quantified by Lipschitz constants of the ICL loss along paths connecting test prompts to pretraining samples, (ii) an intrinsic ICL capability of the pretrained model, and (iii) the degree of distribution shift. Within the same framework, we analyze CoT prompting as inducing a task decomposition and show that it is beneficial when demonstrations are well chosen at each substep and the resulting subtasks are easier to learn. Finally, we characterize how ICL performance sensitivity to prompt templates varies with the number of demonstrations. Together, our study shows that pretraining equips the model with the ability to generalize beyond observed tasks, while CoT enables the model to compose simpler subtasks into more complex ones, and demonstrations and instructions enable it to retrieve similar or complex tasks, including those that can be composed into more complex ones, jointly supporting generalization to unseen tasks. All theoretical insights are corroborated by experiments.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑