Recent advancements in Large Language Models (LLMs) have garnered wide attention and led to successful products such as ChatGPT and GPT-4. Their proficiency in adhering to instructions and delivering harmless, helpful, and honest (3H) responses can largely be attributed to the technique of Reinforcement Learning from Human Feedback (RLHF). In this paper, we aim to link the research in conventional RL to RL techniques used in LLM research. Demystify this technique by discussing why, when, and how RL excels. Furthermore, we explore potential future avenues that could either benefit from or contribute to RLHF research.
Highlighted Takeaways:
1. RLHF is Online Inverse RL with Offline Demonstration Data.
2. RLHF $>$ SFT because Imitation Learning (and Inverse RL) $>$ Behavior Cloning (BC) by alleviating the problem of compounding error.
3. The RM step in RLHF generates a proxy of the expensive human feedback, such an insight can be generalized to other LLM tasks such as prompting evaluation and optimization where feedback is also expensive.
4. The policy learning in RLHF is more challenging than conventional problems studied in IRL due to their high action dimensionality and feedback sparsity.
5. The main superiority of PPO over off-policy value-based methods is its stability gained from (almost) on-policy data and conservative policy updates.
CogBench: a large language model walks into a psychology lab
Max Planck Institute for Biological Cybernetics（马克斯·普朗克生物控制论研究所）；Helmholtz Computational Health Center（亥姆霍兹计算健康中心）；Institute for Human-Centered AI（以人为本人工智能研究所）；Google DeepMind（谷歌DeepMind）
Department of Statistics and Operations Research, UNC-Chapel Hill（统计与运筹学系，北卡罗来纳大学 Chapel Hill 分校）；Imperial College Business School, Imperial College London（帝国理工学院伦敦校区商学院）
HIPO: Instruction Hierarchy via Constrained Reinforcement Learning
School of ECEE Arizona State University（亚利桑那州立大学电子与计算机工程学院）；Department of ECE The Ohio State University（俄亥俄州立大学电子工程系）；Computer Science Department University of Houston（休斯顿大学计算机科学系）；College of Engineering & Applied Science CU Boulder（科罗拉多大学博尔德分校工程与应用科学学院）；Dept. of Electrical Engineering & Computer Science United States Military Academy（美国军事学院电子工程与计算机科学系）
Reprogramming Foundational Large Language Models(LLMs) for Enterprise Adoption for Spatio-Temporal Forecasting Applications: Unveiling a New Era in Copilot-Guided Cross-Modal Time Series Representation Learning
A Comprehensive Survey of Small Language Models in the Era of Large Language Models: Techniques, Enhancements, Applications, Collaboration with LLMs, and Trustworthiness
The Pennsylvania State University（宾夕法尼亚州立大学）；University of Pennsylvania（宾夕法尼亚大学）；UTHealth Houston（德克萨斯大学健康科学中心休斯顿分校）；Amazon（亚马逊公司）；Rensselaer Polytechnic Institute（伦斯勒理工学院）
Explainability for Large Language Models: A Survey
New Jersey Institute of Technology（新泽西理工学院）；Johns Hopkins University（约翰斯·霍普金斯大学）；Wake Forest University（维克森林大学）；University of Georgia（佐治亚大学）；Shanghai Jiao Tong University（上海交通大学）；Institute of Computing Technology, CAS（中国科学院计算技术研究所）；Baidu Inc.（百度公司）；Rice University（莱斯大学）
