Recent work has explored the capability of large language models (LLMs) to identify and correct errors in LLM-generated responses. These refinement approaches frequently evaluate what sizes of models are able to do refinement for what problems, but less attention is paid to what effective feedback for refinement looks like. In this work, we propose looking at refinement with feedback as a composition of three distinct LLM competencies: (1) detection of bad generations; (2) fine-grained natural language critique generation; (3) refining with fine-grained feedback. The first step can be implemented with a high-performing discriminative model and steps 2 and 3 can be implemented either via prompted or fine-tuned LLMs. A key property of the proposed Detect, Critique, Refine ("DCR") method is that the step 2 critique model can give fine-grained feedback about errors, made possible by offloading the discrimination to a separate model in step 1. We show that models of different capabilities benefit from refining with DCR on the task of improving factual consistency of document grounded summaries. Overall, DCR consistently outperforms existing end-to-end refinement approaches and current trained models not fine-tuned for factuality critiquing.
On-Policy Self-Alignment with Fine-grained Knowledge Feedback for Hallucination Mitigation
Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences, Beijing, China(中国科学院软件研究所信息处理实验室,北京,中国);University of Chinese Academy of Sciences, Beijing, China(中国科学院大学,北京,中国);Xiaohongshu Inc(小红书公司)
FRAME: Feedback-Refined Agent Methodology for Enhancing Medical Research Insights
South China University of Technology(南方科技大学);HFIPS, Chinese Academy of Sciences(中国科学院HFIPS);University of Science and Technology of China(中国科学技术大学);The Third Affiliated Hospital of SuYat-sen University(孙逸德大学第三附属医院)
RefCritic: Training Long Chain-of-Thought Critic Models with Refinement Feedback
Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences(中国科学院软件研究所信息处理实验室);Alibaba Group(阿里巴巴集团);University of Chinese Academy of Sciences(中国科学院大学)
Self-Refine: Iterative Refinement with Self-Feedback
Carnegie Mellon University(卡内基梅隆大学);Allen Institute for Artificial Intelligence(艾伦人工智能研究所);University of Washington(华盛顿大学);NVIDIA(英伟达);UC San Diego(加州大学圣地亚哥分校);Google Research(谷歌研究院)
Write, Execute, Refine: From Skill Follower to Skill Optimizer via Reinforcement Learning from Execution Feedback
Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳));The Chinese University of Hong Kong(香港中文大学);Huawei Technologies Co., Ltd.(华为技术有限公司);Harbin Institute of Technology(哈尔滨工业大学)
Cross-Refine: Improving Natural Language Explanation Generation by Learning in Tandem
Technische Universität Berlin(柏林工业大学);Saarland Informatics Campus(萨尔兰信息学园区);Centre for European Research in Trusted AI (CERTAIN)(欧洲可信人工智能研究中心);German Research Center for Artificial Intelligence (DFKI)(德国人工智能研究中心)
Training Language Models with Language Feedback at Scale
New York University(纽约大学);FAR AI;University of the Basque Country UPV/EHU(巴斯克大学);University of Sussex(萨塞克斯大学);Genentech(基因泰克公司);CIFAR LMB(加拿大高级研究院分子生物学与遗传学实验室);Anthropic(Anthropic公司)
Fine-Grained Human Feedback Gives Better Rewards for Language Model Training
University of Washington(华盛顿大学);Allen Institute for Artificial Intelligence(艾伦人工智能研究所);University of California, Berkeley(加州大学伯克利分校);University of California, San Diego(加州大学圣迭戈分校);MosaicML
Beyond Thumbs Up/Down: Untangling Challenges of Fine-Grained Feedback for Text-to-Image Generation
University of Cambridge(剑桥大学);Google DeepMind(谷歌DeepMind);Google Research(谷歌研究院);Field AI;University of Washington(华盛顿大学);University of Tennessee(田纳西大学);Korean Advanced Institute of Science and Technology(韩国高等科学技术院);University of California San Diego(加州大学圣地亚哥分校);The Alan Turing Institute(艾伦·图灵研究所);ServiceNow
LongSumEval: Question-Answering Based Evaluation and Feedback-Driven Refinement for Long Document Summarization
dept. of Information Science University of North Texas Denton, Texas, USA(信息科学系 俄克拉荷马州立大学 德顿, 德克萨斯州, 美国);dept. of Data Science University of North Texas Denton, Texas, USA(数据科学系 俄克拉荷马州立大学 德顿, 德克萨斯州, 美国)
TLCR: Token-Level Continuous Reward for Fine-grained Reinforcement Learning from Human Feedback
Korea Advanced Institute of Science and Technology (KAIST)(韩国科学技术院(KAIST));Kakao Brain;Korea University(高丽大学);University of Illinois at Urbana-Champaign (UIUC)(伊利诺伊大学厄巴纳-香槟分校(UIUC))