Temporal-Difference (TD) learning methods, such as Q-Learning, have proven effective at learning a policy to perform control tasks. One issue with methods like Q-Learning is that the value update introduces bias when predicting the TD target of a unfamiliar state. Estimation noise becomes a bias after the max operator in the policy improvement step, and carries over to value estimations of other states, causing Q-Learning to overestimate the Q value. Algorithms like Soft Q-Learning (SQL) introduce the notion of a soft-greedy policy, which reduces the estimation bias via soft updates in early stages of training. However, the inverse temperature $β$ that controls the softness of an update is usually set by a hand-designed heuristic, which can be inaccurate at capturing the uncertainty in the target estimate. Under the belief that $β$ is closely related to the (state dependent) model uncertainty, Entropy Regularized Q-Learning (EQL) further introduces a principled scheduling of $β$ by maintaining a collection of the model parameters that characterizes model uncertainty. In this paper, we present Unbiased Soft Q-Learning (UQL), which extends the work of EQL from two action, finite state spaces to multi-action, infinite state space Markov Decision Processes. We also provide a principled numerical scheduling of $β$, extended from SQL and using model uncertainty, during the optimization process. We show the theoretical guarantees and the effectiveness of this update method in experiments on several discrete control environments.
University of Science and Technology of China(中国科学技术大学);City University of Hong Kong(香港城市大学);Laboratoire de Physique de l’École Normale Supérieure, Université PSL, CNRS, Sorbonne Université, Université de Paris(巴黎高等师范学院物理实验室,PSL大学,法国国家科学研究中心,索邦大学,巴黎大学)
Efficient Soft Actor-Critic with LLM-Based Action-Level Guidance for Continuous Control
School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China(中国科学院大学人工智能学院,北京,中国);Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China(中国科学院自动化研究所,北京100190,中国)
SoftPipe: A Soft-Guided Reinforcement Learning Framework for Automated Data Preparation
College of Computer Science and Software Engineering, Shenzhen University(深圳大学计算机科学与软件工程学院);School of Artificial Intelligence, Shenzhen University(深圳大学人工智能学院);Graduate School of Information Science and Technology, Osaka University(大阪大学信息科学与技术研究生院)
Mechanism-Guided Selective Unlearning for RLVR-Induced Reasoning
School of Engineering, Institute of Science Tokyo, Japan(东京科学研究院工程学院);College of Control Science and Engineering, Zhejiang University, China(浙江大学控制科学与工程学院);Department of Electrical and Computer Engineering, National University of Singapore, Singapore(新加坡国立大学电子与计算机工程系)
Shanghai Jiao Tong University(上海交通大学);Key Laboratory of System Control and Information Processing, Ministry of Education of China(中国教育部系统控制与信息处理重点实验室);Shanghai Engineering Research Center of Intelligent Control and Management(上海智能控制与管理工程技术研究中心);Shanghai Dianji University(上海电机学院)
CTR-LoRA: Curvature-Aware and Trust-Region Guided Low-Rank Adaptation for Large Language Models
Cornell University(康奈尔大学);University of Alabama at Birmingham(阿拉巴马大学伯明翰分校);Oak Ridge National Laboratory(橡树岭国家实验室);Yale University(耶鲁大学);University of California, Irvine(加州大学欧文分校);Tulane University(路易斯安那大学塔兰帕伊斯分校)
CurvZO: Adaptive Curvature-Guided Sparse Zeroth-Order Optimization for Efficient LLM Fine-Tuning
Department of Computer Science and Engineering, Southern University of Science and Technology, Shenzhen, China.(计算机科学与工程系,南方科技大学,深圳,中国);Department of Mathematics, Southern University of Science and Technology, Shenzhen, China.(数学系,南方科技大学,深圳,中国)
Merging Beyond: Streaming LLM Updates via Activation-Guided Rotations
City University of Hong Kong, Hong Kong SAR(香港城市大学);Tsinghua University(清华大学);Huawei Noah’s Ark Lab, Hong Kong SAR(华为诺亚实验室(香港));University of Hong Kong(香港大学);Hong Kong University of Science and Technology (Guangzhou)(香港理工大学(广州))
Enhancing Pretrained Model-based Continual Representation Learning via Guided Random Projection
Wuhan University(武汉大学);Shanghai Innovation Institute(上海创新研究院);Shanghai Jiao Tong University(上海交通大学);Tsinghua University(清华大学);University of Science and Technology of China(中国科学技术大学);China University of Mining and Technology(中国矿业大学)
Collaborative Optimization of Multiclass Imbalanced Learning: Density-Aware and Region-Guided Boosting
School of Mathematics and Computer, Guangdong Ocean University(数学与计算机学院,广东海洋大学);School of Automation Engineering, University of Electronic Science and Technology of China(自动化工程学院,电子科技大学)