Designing effective reward functions is crucial to training reinforcement learning (RL) algorithms. However, this design is non-trivial, even for domain experts, due to the subjective nature of certain tasks that are hard to quantify explicitly. In recent works, large language models (LLMs) have been used for reward generation from natural language task descriptions, leveraging their extensive instruction tuning and commonsense understanding of human behavior. In this work, we hypothesize that LLMs, guided by human feedback, can be used to formulate reward functions that reflect human implicit knowledge. We study this in three challenging settings -- autonomous driving, humanoid locomotion, and dexterous manipulation -- wherein notions of ``good" behavior are tacit and hard to quantify. To this end, we introduce REvolve, a truly evolutionary framework that uses LLMs for reward design in RL. REvolve generates and refines reward functions by utilizing human feedback to guide the evolution process, effectively translating implicit human knowledge into explicit reward functions for training (deep) RL agents. Experimentally, we demonstrate that agents trained on REvolve-designed rewards outperform other state-of-the-art baselines.
REVOLVE: Optimizing AI Systems by Tracking Response Evolution in Textual Optimization
Hong Kong University of Science and Technology(香港科技大学);University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Brown University(布朗大学);University of Michigan - Ann Arbor(密歇根大学安娜堡分校);Hong Kong Polytechnic University(香港理工大学);Intel Lab(英特尔实验室)
CleanRM: Reward Modeling from Noisy Feedback with a Few Clean Feedback for RLHF
Zhejiang University(浙江大学);Xiaohongshu Inc(小红书公司);National University of Singapore(新加坡国立大学);New York University(纽约大学);University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学)
Using Human Feedback to Fine-tune Diffusion Models without Any Reward Model
Tsinghua Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院);Department of Automation, Tsinghua University(清华大学自动化系);Parametrix Technology Company Ltd.(参数科技有限公司)