Existing policy-gradient methods for auto-regressive language models typically select subsequent tokens one at a time as actions in the policy. While effective for many generation tasks, such an approach may not fully capture the structure of complex reasoning tasks, where a single semantic decision is often realized across multiple tokens--for example, when defining variables or composing equations. This introduces a potential mismatch between token-level optimization and the inherently block-level nature of reasoning in these settings. To bridge this gap, we propose Multi-token Policy Gradient Optimization (MPO), a framework that treats sequences of K consecutive tokens as unified semantic actions. This block-level perspective enables our method to capture the compositional structure of reasoning trajectories and supports optimization over coherent, higher-level objectives. Experiments on mathematical reasoning and coding benchmarks show that MPO outperforms standard token-level policy gradient baselines, highlight the limitations of token-level policy gradients for complex reasoning, motivating future research to look beyond token-level granularity for reasoning-intensive language tasks.
Token-Level Policy Optimization: Linking Group-Level Rewards to Token-Level Aggregation via Sequence-Level Likelihood
College of Computer Science and Technology, Jilin University(吉林大学计算机科学与技术学院);Key Laboratory of Symbolic Computation and Knowledge Engineering of MOE, Jilin University(教育部符号计算与知识工程重点实验室,吉林大学);Baidu Inc.(百度公司);Tsinghua University(清华大学);State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences(中国科学院人工智能安全重点实验室,计算技术研究所)
Token-Level Policy Optimization: Linking Group-Level Rewards to Token-Level Aggregation via Markov Likelihood
Baidu Inc(百度公司);College of Computer Science and Technology, Jilin University(吉林大学计算机科学与技术学院);Key Laboratory of Symbolic Computation and Knowledge Engineering of MOE, Jilin University(吉林大学教育部符号计算与知识工程重点实验室);State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences(中国科学院人工智能安全国家重点实验室)
Enhancing Large Language Model Reasoning via Selective Critical Token Fine-Tuning
Southern University of Science and Technology(南方科技大学);Peking University(北京大学);Shanghai University of Finance and Economics(上海金融学院);Tsinghua University(清华大学)
On the Token Value Inequality in Efficient Reasoning
Purdue University(普渡大学);Rochester Institute of Technology(罗切斯特理工学院);MIT-IBM Watson AI Lab(MIT-IBM沃森人工智能实验室);University of Missouri-Kansas City(密苏里大学堪萨斯城分校);Rutgers University(罗格斯大学);Meta AI
Not All Tokens Matter: Towards Efficient LLM Reasoning via Token Significance in Reinforcement Learning
Tsinghua University(清华大学);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Peking University(北京大学);Qwen Large Model Application Team, Alibaba(阿里云文大模型应用团队);Microsoft(微软);Shenzhen Key Laboratory of Ubiquitous Data Enabling, Tsinghua Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院 ubiquitous 数据赋能重点实验室)
Demystifying Reasoning Dynamics with Mutual Information: Thinking Tokens are Information Peaks in LLM Reasoning
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学北京校区人工智能学院);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);University College London, University of London(伦敦大学大学学院);Dalian University of Technology(大连理工大学)
SelfBudgeter: Adaptive Token Allocation for Efficient LLM Reasoning
State Key Laboratory for Multimedia Information Processing, School of Computer Science, Peking University(多媒体信息处理国家重点实验室,计算机学院,北京大学);BandAI, Bytedance(字节跳动BandAI)
GMTS: Gradient Magnitude-based Token Selection Improves RLVR Training for LLM Reasoning
School of Mathematical Sciences, Shanghai Jiao Tong University(上海交通大学数学科学学院);Institute of Natural Sciences, Shanghai Jiao Tong University(上海交通大学自然科学研究院)
Reasoning-Token Spikes Under Prompted Untruthful Responses in Large Language Models
Chapman University(查普曼大学);INSERM U992, Cognitive Neuroimaging Unit, NeuroSpin Center(法国国家健康与医学研究院U992认知神经影像单元,神经自旋中心);University of California, Los Angeles(加州大学洛杉矶分校);California Institute of Technology(加州理工学院)
Soft Thinking: Unlocking the Reasoning Potential of LLMs in Continuous Concept Space
University of California, Santa Barbara(加州大学圣芭芭拉分校);University of California, Santa Cruz(加州大学圣克ruz分校);University of California, Los Angeles(加州大学洛杉矶分校);Purdue University(普渡大学);LMSYS Org(LMSYS组织);Microsoft(微软)
KTAE: A Model-Free Algorithm to Key-Tokens Advantage Estimation in Mathematical Reasoning
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院)