arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.02366cs.LGcs.AI
收藏

ReasonCACHE: 教授大语言模型进行推理而不进行权重更新

ReasonCACHE: Teaching LLMs To Reason Without Weight Updates

FAIR at Meta(Meta 的 FAIR);MIT CSAIL(麻省理工学院计算机科学与人工智能实验室)

Sharut Gupta, Phillip Isola, Stefanie Jegelka, David Lopez-Paz, Kartik Ahuja, Mark Ibrahim, Mohammad Pezeshki

AI总结:

ReasonCACHE通过前缀微调实现无需权重更新的推理学习,在效率和表现上超越传统ICL和IWL方法。

摘要:

大语言模型(LLMs)能否在不进行任何权重更新的情况下,仅通过上下文学习(ICL)来学习推理?ICL在样本效率方面非常突出,通常仅从少量示例中学习,但复杂的推理任务通常需要许多训练示例才能学习。然而,简单地通过增加更多示例来扩展ICL在这一规模上会失效:注意力成本呈二次增长,性能在更长的上下文中饱和或下降,且该方法仍然是浅层学习形式。由于这些限制,从业者通常依赖于权重学习(IWL)来诱导推理。在本工作中,我们表明通过使用前缀微调,LLMs可以在不超出上下文窗口的情况下学习推理,而无需任何权重更新。我们引入了ReasonCACHE,这是该机制的一个实例,将演示转化为固定键值缓存。实证研究表明,在具有挑战性的推理基准测试中,包括GPQA-Diamond,ReasonCACHE在标准ICL上表现更优,并与或超越IWL方法。此外,它在数据、推理成本和可训练参数三个方面都更加高效。我们还理论证明,ReasonCACHE比低秩权重更新严格更具表达力,因为后者将表达力与输入秩联系起来,而ReasonCACHE通过直接将键值注入注意力机制来绕过这一限制。总的来说,我们的发现表明,ReasonCACHE是上下文学习和权重学习之间的中间路径,提供了一种可扩展的算法,用于在不修改参数的情况下学习超出上下文窗口的推理技能。我们的项目页面:https://reasoncache.github.io/

英文摘要:

Can Large language models (LLMs) learn to reason without any weight update and only through in-context learning (ICL)? ICL is strikingly sample-efficient, often learning from only a handful of demonstrations, but complex reasoning tasks typically demand many training examples to learn from. However, naively scaling ICL by adding more demonstrations breaks down at this scale: attention costs grow quadratically, performance saturates or degrades with longer contexts, and the approach remains a shallow form of learning. Due to these limitations, practitioners predominantly rely on in-weight learning (IWL) to induce reasoning. In this work, we show that by using Prefix Tuning, LLMs can learn to reason without overloading the context window and without any weight updates. We introduce $\textbf{ReasonCACHE}$, an instantiation of this mechanism that distills demonstrations into a fixed key-value cache. Empirically, across challenging reasoning benchmarks, including GPQA-Diamond, ReasonCACHE outperforms standard ICL and matches or surpasses IWL approaches. Further, it achieves this all while being more efficient across three key axes: data, inference cost, and trainable parameters. We also theoretically prove that ReasonCACHE can be strictly more expressive than low-rank weight update since the latter ties expressivity to input rank, whereas ReasonCACHE bypasses this constraint by directly injecting key-values into the attention mechanism. Together, our findings identify ReasonCACHE as a middle path between in-context and in-weight learning, providing a scalable algorithm for learning reasoning skills beyond the context window without modifying parameters. Our project page: https://reasoncache.github.io/

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑