arXiv 2505.133082026-01-19cs.LGcs.AIcs.CL

在黑暗中寻求：通过测试时实例级策略梯度进行潜在空间推理

Seek in the Dark: Reasoning via Test-Time Instance-Level Policy Gradient in Latent Space

Institute for Artificial Intelligence, Peking University（北京大学人工智能研究院）；NLCo Lab, Beijing Institute for General Artificial Intelligence（北京通用人工智能研究院NLCo实验室）；Department of Automation, Tsinghua University（清华大学自动化系）；Shanghai Jiao Tong University（上海交通大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of California, Los Angeles（加州大学洛杉矶分校）

Hengli Li, Chenxi Li, Tong Wu, Xuekai Zhu, Yuxuan Wang, Zhaoxin Yu, Eric Hanchen Jiang, Song-Chun Zhu, Zixia Jia, Ying Nian Wu, Zilong Zheng

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

LatentSeek通过测试时实例级策略梯度在潜在空间中提升LLM推理能力，展现高效且可扩展的性能。

摘要：

推理能力是人类智能的核心组成部分，在追求通用人工智能（AGI）的过程中，大型语言模型（LLMs）在这一方面仍然面临重大挑战。尽管在训练扩展定律下模型性能有所提高，但仍存在显著挑战，尤其是在训练算法方面，如灾难性遗忘，以及新型训练数据的有限可用性。作为一种替代方法，测试时扩展通过增加测试时计算而不更新参数来提高推理性能。与以往在此范式中专注于标记空间的方法不同，我们提出利用潜在空间以获得更有效的推理和更好的遵循测试时扩展定律。我们引入了LatentSeek，一种通过模型潜在空间内的测试时实例级适应（TTIA）来增强LLM推理能力的新框架。具体而言，LatentSeek利用策略梯度来迭代更新潜在表示，受自动生成的奖励信号引导。LatentSeek在多个LLM架构上评估了多种推理基准，包括GSM8K、MATH-500和AIME2024。结果表明，LatentSeek在平均复杂性问题上通常在几次迭代内收敛，同时也能从额外的迭代中受益，从而突显了潜在空间中测试时扩展的潜力。这些发现使LatentSeek成为增强LLM推理能力的一种轻量、可扩展且有效的方法。

英文摘要：

Reasoning ability, a core component of human intelligence, continues to pose a significant challenge for Large Language Models (LLMs) in the pursuit of AGI. Although model performance has improved under the training scaling law, significant challenges remain, particularly with respect to training algorithms, such as catastrophic forgetting, and the limited availability of novel training data. As an alternative, test-time scaling enhances reasoning performance by increasing test-time computation without parameter updating. Unlike prior methods in this paradigm focused on token space, we propose leveraging latent space for more effective reasoning and better adherence to the test-time scaling law. We introduce LatentSeek, a novel framework that enhances LLM reasoning through Test-Time Instance-level Adaptation (TTIA) within the model's latent space. Specifically, LatentSeek leverages policy gradient to iteratively update latent representations, guided by self-generated reward signals. LatentSeek is evaluated on a range of reasoning benchmarks, including GSM8K, MATH-500, and AIME2024, across multiple LLM architectures. Results show that LatentSeek consistently outperforms strong baselines, such as Chain-of-Thought prompting and fine-tuning-based methods. Furthermore, our analysis demonstrates that LatentSeek is highly efficient, typically converging within a few iterations for problems of average complexity, while also benefiting from additional iterations, thereby highlighting the potential of test-time scaling in the latent space. These findings position LatentSeek as a lightweight, scalable, and effective solution for enhancing the reasoning capabilities of LLMs.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。