We propose a new aggregation framework for approximate dynamic programming, which provides a connection with rollout algorithms, approximate policy iteration, and other single and multistep lookahead methods. The central novel characteristic is the use of a bias function $V$ of the state, which biases the values of the aggregate cost function towards their correct levels. The classical aggregation framework is obtained when $V\equiv0$, but our scheme works best when $V$ is a known reasonably good approximation to the optimal cost function $J^*$.
When $V$ is equal to the cost function $J_μ$ of some known policy $μ$ and there is only one aggregate state, our scheme is equivalent to the rollout algorithm based on $μ$ (i.e., the result of a single policy improvement starting with the policy $μ$). When $V=J_μ$ and there are multiple aggregate states, our aggregation approach can be used as a more powerful form of improvement of $μ$. Thus, when combined with an approximate policy evaluation scheme, our approach can form the basis for a new and enhanced form of approximate policy iteration.
When $V$ is a generic bias function, our scheme is equivalent to approximation in value space with lookahead function equal to $V$ plus a local correction within each aggregate state. The local correction levels are obtained by solving a low-dimensional aggregate DP problem, yielding an arbitrarily close approximation to $J^*$, when the number of aggregate states is sufficiently large. Except for the bias function, the aggregate DP problem is similar to the one of the classical aggregation framework, and its algorithmic solution by simulation or other methods is nearly identical to one for classical aggregation, assuming values of $V$ are available when needed.
Make Sparse Rewards Count: Density-Aware Reward Aggregation for Multi-Reward RL
University of Chinese Academy of Sciences（中国科学院大学）；University of Minnesota Twin Cities（明尼苏达大学双城分校）；University of Wisconsin–Madison（威斯康星大学麦迪逊分校）；Stony Brook University（纽约州立大学石溪分校）；Dalian University of Technology（大连理工大学）；Beijing Foreign Studies University（北京外国语大学）；Southeast University（东南大学）；Kuaishou Technology（快手科技）
Model-Based Offline Reinforcement Learning with Adversarial Data Augmentation
National Key Laboratory for Novel Software Technology, Nanjing University（南京大学现代软件技术国家重点实验室）；City University of Hong Kong（香港城市大学）；School of Computer Science, Nanjing University of Posts and Telecommunications（南京邮电大学计算机学院）；Department of Computer Science, University of Liverpool（利物浦大学计算机系）
Multilingual Verifier Bias in RLVR: Benchmark, Rollout Diagnosis, and the Cross-Lingual Selection Bottleneck
School of Engineering, Institute of Science Tokyo（科学东京大学工程学院）；College of Control Science and Engineering, Zhejiang University（浙江大学控制科学与工程学院）；Department of Electrical and Computer Engineering, National University of Singapore（新加坡国立大学电气与计算机工程学院）
Université Libre de Bruxelles（布鲁塞尔自由大学）；Vrije Universiteit Brussel（布鲁塞尔自由大学）；UC Berkeley（加州大学伯克利分校）；National Research Council（国家研究委员会）
Quite some real-world problems can be formulated as decision-making problems wherein one must repeatedly make an appropriate choice from a set of alternatives. Multiple expert judgements, whether human or artificial, can help in taking correct decisions, especially when exploration of alternative solutions is costly. As expert opinions might deviate, the problem of finding the right alternative can be approached as a collective decision making problem (CDM) via aggregation of independent judgements. Current state-of-the-art approaches focus on efficiently finding the optimal expert, and thus p…
ADHint: Adaptive Hints with Difficulty Priors for Reinforcement Learning
Alibaba Group（阿里巴巴集团）；Beijing Institute of Technology（北京理工大学）；Peking University（北京大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Zhongguancun Academy（中关村学院）
Compute as Teacher: Turning Inference Compute Into Reference-Free Supervision
University of Oxford（牛津大学）；ELLIS Institute Tübingen（图宾根ELLIS研究所）；Max Planck Institute for Intelligent Systems（智能系统马克斯·普朗克研究所）；Meta Superintelligence Labs（元宇宙超级智能实验室）
GD$^2$PO: Mitigating Multi-Reward Conflicts via Group-Dynamic reward-Decoupled Policy Optimization
Qwen Large Model Application Team, Alibaba（阿里巴巴通义千问大模型应用团队）；Renmin University of China（中国人民大学）；Peking University（北京大学）；ETH Zürich（苏黎世联邦理工学院）；University of Zurich（苏黎世大学）；The Chinese University of Hong Kong（香港中文大学）
Federated learning has emerged as an innovative paradigm of collaborative machine learning. Unlike conventional machine learning, a global model is collaboratively learned while data remains distributed over a tremendous number of client devices, thus not compromising user privacy. However, several challenges still remain despite its glowing popularity; above all, the global aggregation in federated learning involves the challenge of biased model averaging and lack of prior knowledge in client sampling, which, in turn, leads to high generalization error and slow convergence rate, respectively.…
Advantage-level Aggregation Reinforcement Learning for X-point Target Magnetic Configuration Control in an EXL-50U Experiment-Calibrated Simulation Environment
Beijing ENN Fusion Energy Science and Technology Co., Ltd.（北京新奥融合能源科技有限公司）；Beijing Key Laboratory of High Magnetic Field Spherical Torus Fusion Energy（北京强磁场球形托卡马克聚变能源重点实验室）；Hebei Key Laboratory of Compact Fusion（河北省紧凑型聚变重点实验室）；Dalian University of Technology（大连理工大学）；School of Physics, Dalian University of Technology（大连理工大学物理学院）；Key Laboratory of Materials Modification by Beams of the Ministry of Education（教育部材料表面与界面工程重点实验室）；School of Nuclear Science and Engineering, East China University of Technology（东华理工大学核科学与工程学院）；East China University of Technology（东华理工大学）；School of Mechanics and Engineering Science, Peking University（北京大学力学与工程科学学院）；Peking University（北京大学）
University of Pennsylvania（宾夕法尼亚大学）；JPMorgan AI Research（摩根大通人工智能研究院）
Reinforcement learning, mathematically described by Markov Decision Problems, may be approached either through dynamic programming or policy search. Actor-critic algorithms combine the merits of both approaches by alternating between steps to estimate the value function and policy gradient updates. Due to the fact that the updates exhibit correlated noise and biased gradient updates, only the asymptotic behavior of actor-critic is known by connecting its behavior to dynamical systems. This work puts forth a new variant of actor-critic that employs Monte Carlo rollouts during the policy search …
On the Push-Based Asynchronous Federated Learning: A Bias-Correction Aggregation Approach
School of Computer Technologies, RMIT University（RMIT大学计算机技术学院）；School of Science, Computing and Engineering Technologies, Swinburne University of Technology（斯威丁大学科学与工程技术学院）
Demystifying OPD: Length Inflation and Stabilization Strategies for Large Language Models
Department of Computer Science, Rice University, Houston, USA（莱斯大学计算机科学系，美国休斯顿）；Department of Computer Science, University of North Carolina at Charlotte, Charlotte, USA（北卡罗来纳大学夏洛特分校计算机科学系，美国夏洛特）；Department of Computer Science, Johns Hopkins University, Baltimore, USA（约翰霍普金斯大学计算机科学系，美国巴尔的摩）；Department of Computer and Data Sciences, Case Western Reserve University（凯斯西储大学计算机与数据科学系）
Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixes
State Key Laboratory of Multimodal Artificial Intelligence Systems, CASIA（多模态人工智能系统国家重点实验室，中科院自动化所）；School of Artificial Intelligence, UCAS（中国科学技术大学人工智能学院）；Fudan University（复旦大学）；Independent Researcher（独立研究者）
In practice, and especially when training deep neural networks, visual recognition rules are often learned based on various sources of information. On the other hand, the recent deployment of facial recognition systems with uneven performances on different population segments has highlighted the representativeness issues induced by a naive aggregation of the datasets. In this paper, we show how biasing models can remedy these problems. Based on the (approximate) knowledge of the biasing mechanisms at work, our approach consists in reweighting the observations, so as to form a nearly debiased e…
Breaking the Prototype Bias Loop: Confidence-Aware Federated Contrastive Learning for Highly Imbalanced Clients
Key Laboratory of Water Big Data Technology of Ministry of Water Resources, College of Computer Science and Software Engineering, Hohai University, Nanjing, China（水利部水大数据技术重点实验室，计算机科学与软件工程学院，河海大学，南京，中国）；Department of Computer Science, City University of Hong Kong, Hong Kong, China（计算机科学系，香港城市大学，香港，中国）；State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China（新型软件技术国家重点实验室，南京大学，南京，中国）；School of Artificial Intelligence and Information Technology, Nanjing University of Chinese Medicine, Nanjing, China（人工智能与信息技术学院，南京中医药大学，南京，中国）
