Large language models (LLMs) can be improved by aligning with human preferences through fine-tuning -- the so-called reinforcement learning from human feedback (RLHF). However, the cost of fine-tuning an LLM is prohibitive for many users. Due to their ability to bypass LLM fine-tuning, prediction-time tokenwise reward-guided text generation (RGTG) methods have recently been proposed. They use a reward model trained on full sequences to score partial sequences during decoding in a bid to steer the generation towards sequences with high rewards. However, these methods have so far been only heuristically motivated and poorly analyzed. In this work, we show that reward models trained on full sequences are not compatible with scoring partial sequences. To alleviate this, we propose to train a Bradley-Terry reward model on partial sequences explicitly, and autoregressively sample from the implied tokenwise policy during decoding. We study the properties of this reward model and the resulting policy: we show that this policy is proportional to the ratio of two distinct RLHF policies. Our simple approach outperforms previous RGTG methods and performs similarly to strong offline baselines without large-scale LLM fine-tuning.
Code for our work is available at https://github.com/ahmadrash/PARGS
EAGER: Asking and Answering Questions for Automatic Reward Shaping in Language-guided RL
Inria(法国国家信息与自动化研究所);Université de Bordeaux(波尔多大学);ISIR Sorbonne Université(索邦大学智能系统与机器人研究所);Univ Angers(昂热大学);Microsoft Research Montreal(微软研究院蒙特利尔分院)
Self-Guided Process Reward Optimization with Redefined Step-wise Advantage for Process Reinforcement Learning
Terminus Group(Terminus集团);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));King Abdullah University of Science and Technology(国王阿卜杜勒阿齐兹大学)
LinguaFluid: Language Guided Fluid Control via Semantic Rewards in Reinforcement Learning
College of Environmental and Resource Sciences, Zhejiang University, Hangzhou, China(浙江大学环境与资源学院);School of Engineering, Westlake University, Hangzhou, China(西湖大学工程学院)
OMG-RL:Offline Model-based Guided Reward Learning for Heparin Treatment
Department of Industrial and Information Systems Engineering, Soongsil University(工业与信息系统工程系,顺斯大学);Department of Systems Management Engineering, Sungkyunkwan University(系统管理工程系,成均馆大学)
Potential Field Guided Actor-Critic Reinforcement Learning
Artificial Intelligence Research Center of National Innovation Institute of Defense Technology(国防科技创新研究院人工智能研究中心);Tianjin Artificial Intelligence Innovation Center(天津人工智能创新中心)
RAPiD: Reward-Guided Consistency Distillation of Diffusion Planners for Real-Time Autonomous Driving
School of Information Technology, Monash University, Malaysia(墨尔本大学信息科技学院);Faculty of Information Technology, Monash University, Australia(墨尔本大学信息技术学院);TCS Research, India(印度TCS研究)
TR2-D2: Tree Search Guided Trajectory-Aware Fine-Tuning for Discrete Diffusion
Department of Computer and Information Science, University of Pennsylvania(计算机与信息科学系,宾夕法尼亚大学);School of Mathematics, Georgia Institute of Technology(数学学院,佐治亚理工学院);Department of Bioengineering, University of Pennsylvania(生物工程系,宾夕法尼亚大学)
Learning in Context, Guided by Choice: A Reward-Free Paradigm for Reinforcement Learning with Transformers
Department of Electrical and Computer Engineering, Duke University, Durham, US(电子工程系,杜克大学,达勒姆,美国);Department of Computer Science, Duke University, Durham, US(计算机科学系,杜克大学,达勒姆,美国);Department of Biostatistics and Bioinformatics, Duke University, Durham, US(生物统计学与生物信息学系,杜克大学,达勒姆,美国);Department of Statistics and Data Science, Yale University, New Haven, US(统计学与数据科学系,耶鲁大学,纽 Haven,美国)
Graduate School of Computer Science, Saarland University(萨尔大学计算机科学研究生院);Bosch Center for Artificial Intelligence, Robert Bosch GmbH(罗伯特·博世有限公司博世人工智能中心);CISPA Helmholtz Center for Information Security(CISPA亥姆霍兹信息安全中心)
Hybrid Energy-Aware Reward Shaping: A Unified Lightweight Physics-Guided Methodology for Policy Optimization
School of Mechanical Engineering, University of Science and Technology Beijing(北京科技大学机械工程学院);Jiangsu XCMG Construction Machinery Research Institute Co., Ltd.(江苏中联重科工程机械研究院有限公司)
Coordinating Ride-Pooling with Public Transit using Reward-Guided Conservative Q-Learning: An Offline Training and Online Fine-Tuning Reinforcement Learning Framework
The Hong Kong University of Science and Technology(香港科技大学);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州))