Kernelized Advantage Estimation: From Nonparametric Statistics to LLM Reasoning
School of Management, University of Science and Technology of China（中国科学技术大学管理学院）；Department of Statistics, London School of Economics and Political Science（伦敦政治经济学院统计系）；School of Mathematics, University of Birmingham（伯明翰大学数学学院）；Academy of Mathematics and Systems Science, Chinese Academy of Sciences（中国科学院数学与系统科学研究院）；Department of Mathematics, Tsinghua University（清华大学数学系）
Shijin Gong, Kai Ye, Jin Zhu, Xinyu Zhang, Hongyi Zhou, Chengchun Shi
Recent advances in large language models (LLMs) have increasingly relied on reinforcement learning (RL) to improve their reasoning capabilities. Three types of approaches have been widely adopted: The first relies on a deep neural network to estimate the value function of the learning policy in order to reduce the variance of the policy gradient. However, estimating and maintaining such a value network incurs substantial computational and memory overhead. The second avoids training a value network by approximating the value function using sample averages. However, it samples a large number of reasoning traces per prompt for accurate value function approximation, making it computationally expensive. The third samples only a single reasoning trajectory per prompt, which reduces computational cost but suffers from poor sample efficiency.
This paper focuses on a practical, resource-constrained setting in which only a small number of reasoning traces can be sampled per prompt, while low-variance gradient estimation remains essential for high-quality policy learning. To address this challenge, we bring classical nonparametric statistical methods, which are both computationally and statistically efficient, to LLM reasoning. We employ kernel smoothing as a concrete example for value function estimation and the subsequent policy optimization. Numerical and theoretical results demonstrate that our proposal achieves accurate value and gradient estimation, leading to improved policy optimization.
Online nonparametric regression with Sobolev kernels
University of Potsdam（波茨坦大学）；Centre de Recherche INRIA de Paris（法国国家信息与自动化研究所巴黎研究中心）；IRT Saint Exupéry（圣埃克苏佩里技术研究院）；Institut de Mathématiques de Toulouse（图卢兹数学研究所）
