Fine-Grained Human Feedback Gives Better Rewards for Language Model Training
University of Washington(华盛顿大学);Allen Institute for Artificial Intelligence(艾伦人工智能研究所);University of California, Berkeley(加州大学伯克利分校);University of California, San Diego(加州大学圣迭戈分校);MosaicML
Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A. Smith, Mari Ostendorf, Hannaneh Hajishirzi
Language models (LMs) often exhibit undesirable text generation behaviors, including generating false, toxic, or irrelevant outputs. Reinforcement learning from human feedback (RLHF) - where human preference judgments on LM outputs are transformed into a learning signal - has recently shown promise in addressing these issues. However, such holistic feedback conveys limited information on long text outputs; it does not indicate which aspects of the outputs influenced user preference; e.g., which parts contain what type(s) of errors. In this paper, we use fine-grained human feedback (e.g., which sentence is false, which sub-sentence is irrelevant) as an explicit training signal. We introduce Fine-Grained RLHF, a framework that enables training and learning from reward functions that are fine-grained in two respects: (1) density, providing a reward after every segment (e.g., a sentence) is generated; and (2) incorporating multiple reward models associated with different feedback types (e.g., factual incorrectness, irrelevance, and information incompleteness). We conduct experiments on detoxification and long-form question answering to illustrate how learning with such reward functions leads to improved performance, supported by both automatic and human evaluation. Additionally, we show that LM behaviors can be customized using different combinations of fine-grained reward models. We release all data, collected human feedback, and codes at https://FineGrainedRLHF.github.io.
Fine-Grained Reward Optimization for Machine Translation using Error Severity Mappings
Instituto Superior Técnico, Universidade de Lisboa (ELLIS Unit Lisbon)(里斯本大学理工学院(ELLIS单位里斯本));Instituto de Telecomunicações(电信研究所);Carnegie Mellon University(卡内基梅隆大学);TransPerfect Core contributor(TransPerfect核心贡献者)
From Coarse to Fine: Benchmarking and Reward Modeling for Writing-Centric Generation Tasks
Shanghai Key Laboratory of Data Science, College of Computer Science and Artificial Intelligence, Fudan University(数据科学上海重点实验室,计算机科学与人工智能学院,复旦大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室)
TLCR: Token-Level Continuous Reward for Fine-grained Reinforcement Learning from Human Feedback
Korea Advanced Institute of Science and Technology (KAIST)(韩国科学技术院(KAIST));Kakao Brain;Korea University(高丽大学);University of Illinois at Urbana-Champaign (UIUC)(伊利诺伊大学厄巴纳-香槟分校(UIUC))
The Flip Side of RLHF: On-Policy Feedback for Reward Model Self-Supervised Improvement
State Key Laboratory of Cognitive Intelligence, University of Science and Technology of China(认知智能国家重点实验室,中国科学技术大学);University of Science and Technology of China(中国科学技术大学);Institute of Artificial Intelligence, Hefei Comprehensive National Science Center(合肥综合性国家科学中心人工智能研究院);State Key Laboratory of General Artificial Intelligence, BIGAI(通用人工智能国家重点实验室,BIGAI)
On-Policy Self-Alignment with Fine-grained Knowledge Feedback for Hallucination Mitigation
Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences, Beijing, China(中国科学院软件研究所信息处理实验室,北京,中国);University of Chinese Academy of Sciences, Beijing, China(中国科学院大学,北京,中国);Xiaohongshu Inc(小红书公司)
RewardExplainer: Learning Reward Model Explanations from Counterfactual Preference Feedback
The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Shanghai Jiao Tong University(上海交通大学);Carnegie Mellon University(卡内基梅隆大学);Jilin University(吉林大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室)
Beyond Majority Voting: Towards Fine-grained and More Reliable Reward Signal for Test-Time Reinforcement Learning
College of Computer Science and Software Engineering, Shenzhen University(深圳大学计算机科学与软件工程学院);College of Computer and Data Science, Fuzhou University(福州大学计算机与数据科学学院)
Beyond Thumbs Up/Down: Untangling Challenges of Fine-Grained Feedback for Text-to-Image Generation
University of Cambridge(剑桥大学);Google DeepMind(谷歌DeepMind);Google Research(谷歌研究院);Field AI;University of Washington(华盛顿大学);University of Tennessee(田纳西大学);Korean Advanced Institute of Science and Technology(韩国高等科学技术院);University of California San Diego(加州大学圣地亚哥分校);The Alan Turing Institute(艾伦·图灵研究所);ServiceNow
PRMBench: A Fine-grained and Challenging Benchmark for Process-Level Reward Models
Fudan University(复旦大学);Shanghai AI Laboratory(上海人工智能实验室);Soochow University(苏州大学);Stony Brook University(石溪大学);The Chinese University of Hong Kong(香港中文大学)