Despite scale driving substantial recent advancements in machine learning, reinforcement learning (RL) methods still primarily use small value functions. Naively scaling value functions -- including with a transformer architecture, which is known to be highly scalable -- often results in learning instability and worse performance. In this work, we ask what prevents transformers from scaling effectively for value functions? Through empirical analysis, we identify the critical failure mode in this scaling: attention scores collapse as capacity increases. Our key insight is that we can effectively prevent this collapse and stabilize training by controlling the entropy of the attention scores, thereby enabling the use of larger models. To this end, we propose Transformer Q-Learning (TQL), a method that unlocks the scaling potential of transformers in learning value functions in RL. Our approach yields up to a 43% improvement in performance when scaling from the smallest to the largest network sizes, while prior methods suffer from performance degradation.
Preventing Model Collapse via Contraction-Conditioned Neural Filters
School of Mathematical Sciences, Tonji University, Shanghai, 200092, P.\,R. China；School of Mathematical Sciences, Nankai University, Tianjin, China；Independent Researcher；Longhua Clinical College, Shanghai University of Traditional Chinese Medicine, Shanghai, China；Dundee International Institute, Central South University, Changsha, China；School of Mathematical Sciences, Shanghai Jiao Tong University, Shanghai, China；School of Life Sciences；Biotechnology, Shanghai Jiao Tong University, Shanghai, China
Beyond Mode Collapse: Distribution Matching for Diverse Reasoning
Tongji University（同济大学）；Independent（独立）；Shanghai AI Laboratory（上海人工智能实验室）；Zhejiang University（浙江大学）；Fudan University（复旦大学）；The Chinese University of Hong Kong（香港中文大学）
Two is better than one: A Collapse-free Multi-Reward RLIF Training Framework
Bangladesh University of Engineering and Technology（孟加拉工程科技大学）；West Virginia University（西弗吉尼亚大学）；University of Aberdeen（阿伯丁大学）；Fogsphere (Redev.AI Ltd, UK)（Fogsphere（Redev.AI Ltd，英国））；University College London（伦敦大学学院）
Stabilizing MoE Reinforcement Learning by Aligning Training and Inference Routers
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University（多媒体信息处理国家重点实验室，计算机学院，北京大学）；LLM-Core Xiaomi（小米LLM-Core）
Balancing policy constraint and ensemble size in uncertainty-based offline reinforcement learning
Warwick Medical School, University of Warwick（华威大学华威医学院）；WMG, University of Warwick（华威大学华威制造集团）；Department of Statistics, University of Warwick（华威大学统计系）；Alan Turing Institute（艾伦·图灵研究所）
