It is often desirable for Large Language Models (LLMs) to capture multiple objectives when providing a response. In document-grounded response generation, for example, agent responses are expected to be relevant to a user's query while also being grounded in a given document. In this paper, we introduce Proxy Metric-based Self-Refinement (ProMiSe), which enables an LLM to refine its own initial response along key dimensions of quality guided by external metrics feedback, yielding an overall better final response. ProMiSe leverages feedback on response quality through principle-specific proxy metrics, and iteratively refines its response one principle at a time. We apply ProMiSe to open source language models Flan-T5-XXL and Llama-2-13B-Chat, to evaluate its performance on document-grounded question answering datasets, MultiDoc2Dial and QuAC, demonstrating that self-refinement improves response quality. We further show that fine-tuning Llama-2-13B-Chat on the synthetic dialogue data generated by ProMiSe yields significant performance improvements over the zero-shot baseline as well as a supervised fine-tuned model on human annotated data.
Self-Refine: Iterative Refinement with Self-Feedback
Carnegie Mellon University(卡内基梅隆大学);Allen Institute for Artificial Intelligence(艾伦人工智能研究所);University of Washington(华盛顿大学);NVIDIA(英伟达);UC San Diego(加州大学圣地亚哥分校);Google Research(谷歌研究院)
Proxy-RLHF: Decoupling Generation and Alignment in Large Language Model with Proxy
University of Science and Technology of China(中国科学技术大学);Deep Space Exploration Laboratory(深空探测实验室);Institute for Advanced Algorithms Research(先进算法研究院);Sangfor Technologies Inc.(深信服科技股份有限公司);Research Institute of China Telecom(中国电信研究院)
Training Language Models with Language Feedback at Scale
New York University(纽约大学);FAR AI;University of the Basque Country UPV/EHU(巴斯克大学);University of Sussex(萨塞克斯大学);Genentech(基因泰克公司);CIFAR LMB(加拿大高级研究院分子生物学与遗传学实验室);Anthropic(Anthropic公司)
A Pair Programming Framework for Code Generation via Multi-Plan Exploration and Feedback-Driven Refinement
State Key Laboratory for Novel Software Technology(计算机软件新技术国家重点实验室);Nanjing University(南京大学);National Institute of Healthcare Data Science(国家卫生健康数据科学研究院)
RefCritic: Training Long Chain-of-Thought Critic Models with Refinement Feedback
Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences(中国科学院软件研究所信息处理实验室);Alibaba Group(阿里巴巴集团);University of Chinese Academy of Sciences(中国科学院大学)
Effective LLM Code Refinement via Property-Oriented and Structurally Minimal Feedback
School of Software, Beihang University, Beijing, China(北京航空航天大学软件学院);Shanghai AI Laboratory, Shanghai, China(上海人工智能实验室);Shanghai Innovation Institute, Shanghai, China(上海创新研究院)
OpenCodeInterpreter: Integrating Code Generation with Execution and Refinement
Multimodal Art Projection Research Community(多模态艺术投影研究共同体);University of Waterloo(滑铁卢大学);Allen Institute for Artificial Intelligence(艾伦人工智能研究所);HKUST(香港科技大学);IN.AI Research(IN.AI研究机构)
MaxSAT-Based Feedback for Guiding Vision-Language Models in Sudoku
Artificial Intelligence Research Institute (IIIA), Consejo Superior de Investigaciones Científicas (CSIC)(人工智能研究所(IIIA),西班牙科学研究高级理事会(CSIC));Institut de Robòtica i Informàtica Industrial (IRI-CSIC-UPC)(工业机器人与信息学研究所(IRI-CSIC-UPC))
Wasserstein Distributionally Robust Regret Optimization for Reinforcement Learning from Human Feedback
Department of Statistics and Operations Research, University of North Carolina(统计与运筹学系,北卡罗来纳大学);Imperial Business School, Imperial College London(帝国理工学院伦敦商学院);Department of Management Science and Engineering, Stanford University(管理科学与工程系,斯坦福大学)
Training Language Model to Critique for Better Refinement
China Telecom Research Institute(中国电信研究院);The Conversational Artificial Intelligence (CoAI) Group, Tsinghua University(清华大学对话人工智能(CoAI)小组);University of Electronic Science and Technology of China(电子科技大学);The Knowledge Engineering Group (KEG), Tsinghua University(清华大学知识工程小组);Zhipu AI(智谱AI)
Verification and Refinement of Natural Language Explanations through LLM-Symbolic Theorem Proving
University of Manchester(曼彻斯特大学);Idiap Research Institute(Idiap研究所);National Biomarker Centre, CRUK-MI, University of Manchester(英国曼彻斯特大学英国癌症研究中心曼彻斯特研究所国家生物标志物中心)