Reinforcement-learning-based post-training has become a key approach for improving the reasoning ability of large language models, but its token-level learning signals remain poorly understood. This work studies their heterogeneity through attention entropy, which measures how concentrated or diffuse the contextual support is for each response token.
We first show that token-level RL objectives are sparsely estimable: uniformly random 20 percent token subsets preserve much of the full-token held-out performance, suggesting substantial redundancy in token-level updates. However, entropy-structured subsets behave very differently. Low-attention-entropy tokens, which we call anchors, rely on concentrated support, produce stable gradients aligned with full-token updates, and provide a reliable optimization backbone, but tend to plateau on harder benchmarks. High-attention-entropy tokens, which we call explorers, aggregate more diffuse context and induce larger but more volatile gradients. Explorer-only training is unstable on average, though rare successful runs suggest that these tokens may contain useful hard-reasoning signals when optimization remains stable.
We support this anchor-explorer spectrum with evidence-gathering analyses, entropy dynamics, gradient-geometry diagnostics, and controls showing that position, predictive entropy, and loss normalization do not explain the observed asymmetry. Finally, a dynamic entropy-aware soft-reweighting intervention improves Qwen3-8B-Base from 34.39 to 37.40 held-out average in the strongest setting. These findings suggest that attention entropy reveals optimization-relevant structure in token-level RL signals, and that uniform token averaging can obscure meaningful heterogeneity in reasoning post-training.
Heterogeneous Adaptive Policy Optimization: Tailoring Optimization to Every Token's Nature
Peking University Shanghai AI Laboratory(北京大学上海人工智能实验室);Beihang University(北京航空航天大学);Shanghai AI Laboratory(上海人工智能实验室);Peking University Zhongguancun Academy, 5 Beijing Key Laboratory of Software and Hardware Cooperative Artificial Intelligence Systems(北京大学中关村学院,5北京软件硬件协同人工智能系统重点实验室)
Masked Language Models are Good Heterogeneous Graph Generalizers
School of Computer Science, Beijing University of Posts and Telecommunications(北京邮电大学计算机科学学院);Department of Computer Science, University of Illinois Chicago(伊利诺伊大学芝加哥分校计算机科学系);Institute for Artificial Intelligence, Peking University(北京大学人工智能研究院);Ant Group, Beijing, China(蚂蚁集团(北京))
Institutional Books 1.0: A 242B token dataset from Harvard Library's collections, refined for accuracy and usability
Harvard Law School Library(哈佛法学院图书馆);Harvard Library(哈佛大学图书馆);Harvard Law School(哈佛法学院);Harvard School of Engineering and Applied Sciences(哈佛大学工程与应用科学学院);Harvard Kennedy School(哈佛肯尼迪学院)
Mind the Gap No More: Achieving Zero-Gap Multimodal Integration via One Tokenizer
Research Center for Frontier Fundamental Studies, Zhejiang Lab(前沿基础研究研究中心,浙江实验室);Research Center for Scientific Data Hub, Zhejiang Lab(科学数据枢纽研究中心,浙江实验室)
Mixture of Weight-shared Heterogeneous Group Attention Experts for Dynamic Token-wise KV Optimization
University of Chinese Academy of Sciences(中国科学院大学);Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences(中国科学院深圳先进技术研究院);University of Macau(澳门大学);Shenzhen University of Advanced Technology(深圳先进技术大学)