arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2604.01430cs.LG
收藏

通过测试时间计算提升潜在泛化能力

Improving Latent Generalization Using Test-time Compute

Google DeepMind(谷歌DeepMind)

Arslan Chaudhry, Sridhar Thiagarajan, Andrew Lampinen

AI总结:

本文研究如何通过测试时间计算('思考')提升语言模型的潜在泛化能力,利用强化学习训练模型生成长链思考以改进泛化,实验显示该方法在分布内知识和分布外知识上均有效,但对纯反转任务仍逊于上下文学习。

摘要:

语言模型(LMs)在知识获取方面表现出两种不同的机制:权重内学习(即在模型权重中编码信息)和上下文学习(ICL)。尽管这两种模式具有互补的优势,但权重内学习在促进内部知识的演绎推理方面经常遇到困难。我们将这种限制称为潜在泛化能力的不足,其中反转诅咒是一个例子。相反,上下文学习表现出高度稳健的潜在泛化能力。为了提高从权重内知识的潜在泛化能力,先前的方法依赖于训练时间数据增强,但这些技术任务特定、扩展性差,无法泛化到分布外知识。为了克服这些不足,本文研究了模型如何被教导利用测试时间计算,即'思考',特别是以改进潜在泛化能力。我们使用来自正确性反馈的强化学习(RL)来训练模型生成长链思考(CoTs)以改进潜在泛化。我们的实验显示,这种思考方法不仅解决了许多分布内知识的潜在泛化失败实例,而且不同于增强基线,能够泛化到没有RL进行的新知识。然而,在纯反转任务上,我们发现思考无法解锁直接的知识反转,但思考模型的生成和验证能力使它们能够超越偶然表现。事实性的自我验证的脆弱性意味着思考模型在此任务上仍低于上下文学习的性能。总体而言,我们的结果确立了测试时间思考作为改进LM潜在泛化能力的灵活且有前途的方向。

英文摘要:

Language Models (LMs) exhibit two distinct mechanisms for knowledge acquisition: in-weights learning (i.e., encoding information within the model weights) and in-context learning (ICL). Although these two modes offer complementary strengths, in-weights learning frequently struggles to facilitate deductive reasoning over the internalized knowledge. We characterize this limitation as a deficit in latent generalization, of which the reversal curse is one example. Conversely, in-context learning demonstrates highly robust latent generalization capabilities. To improve latent generalization from in-weights knowledge, prior approaches rely on train-time data augmentation, yet these techniques are task-specific, scale poorly, and fail to generalize to out-of-distribution knowledge. To overcome these shortcomings, this work studies how models can be taught to use test-time compute, or 'thinking', specifically to improve latent generalization. We use Reinforcement Learning (RL) from correctness feedback to train models to produce long chains-of-thought (CoTs) to improve latent generalization. Our experiments show that this thinking approach not only resolves many instances of latent generalization failures on in-distribution knowledge but also, unlike augmentation baselines, generalizes to new knowledge for which no RL was performed. Nevertheless, on pure reversal tasks, we find that thinking does not unlock direct knowledge inversion, but the generate-and-verify ability of thinking models enables them to get well above chance performance. The brittleness of factual self-verification means thinking models still remain well below the performance of in-context learning for this task. Overall, our results establish test-time thinking as a flexible and promising direction for improving the latent generalization of LMs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑