上下文学习中思维链的渐近理论
An Asymptotic Theory of Chain-of-Thought in In-Context Learning
AI总结：
通过高维随机矩阵理论，推导了线性回归中上下文学习思维链的泛化误差精确公式，揭示了推理深度、预训练数据量和上下文长度之间的相变现象。
摘要：
思维链推理已成为一种广泛使用的机制，通过在推理时生成中间推理步骤来激发大型语言模型的多步推理。然而，泛化能力随思维链深度的缩放行为仍知之甚少。为了解决这个问题，我们研究了一个理论上可解的线性回归中上下文权重预测的思维链模型，其中测试时推理表示为权重参数估计的迭代细化。利用高维渐近下的随机矩阵理论工具，我们推导了泛化误差作为推理深度、预训练数据量和上下文长度的精确公式。我们的分析揭示了指数与多项式改进、饱和及过度思考之间的尖锐相变，并刻画了最优推理深度如何缩放。我们进一步表明，更深的推理在预训练和上下文信息足够丰富时最为有效，而有限的预训练或上下文会使较长的推理容易产生误差放大或饱和。我们还通过在完全学习的线性注意力和softmax注意力模型上的实验验证了这些预测。我们的结果为测试时思维链深度如何影响泛化提供了一个统一的理论解释。
英文摘要：
Chain-of-thought (CoT) reasoning has become a widely used mechanism for eliciting multi-step reasoning in large language models by generating intermediate reasoning steps at inference time. Yet the scaling behavior of generalization with CoT depth remains poorly understood. To address this question, we study a theoretically solvable model of CoT for in-context weight prediction in linear regression, where test-time reasoning is represented as an iterative refinement of the weight-parameter estimate. Using tools from random matrix theory under high-dimensional asymptotics, we derive an exact formula for the generalization error as a function of reasoning depth, pretraining data amount, and context length. Our analysis reveals a sharp phase transition separating exponential and polynomial improvement, saturation, and overthinking, and characterizes how the optimal reasoning depth scales. We further show that deeper reasoning is most effective with sufficiently rich pretraining and in-context information, whereas limited pretraining or context makes longer reasoning prone to error amplification or saturation. We also validate these predictions through experiments on fully learned linear attention and softmax attention models. Our results provide a unified theoretical account of how test-time CoT depth affects generalization.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
