大语言模型(LLMs)已在诸多领域展现出卓越的通用性。为进一步推动LLMs发展,我们提出了SELF(Self-Evolution with Language Feedback,基于语言反馈的自我进化)这一全新方法,它能让LLMs像人类学习过程一样通过自我反思实现自我提升。SELF首先启动元技能学习流程,让LLMs具备自我反馈和自我优化的能力。随后,模型进入迭代式自我进化阶段:每一轮迭代中,模型利用无标注指令数据集生成初始响应,再通过自我反馈和自我优化对这些响应进行增强,之后用增强后的数据对模型进行微调,模型通过这一迭代自我进化过程实现逐步提升。此外,SELF框架支持模型在推理阶段应用自我优化,进一步提升响应质量。我们在数学任务和通用任务上开展的实验表明,SELF无需人工干预即可增强LLMs的能力。SELF框架为LLMs的自主进化指明了极具前景的方向,推动其从被动的信息接收者转变为自身发展过程中的主动参与者。
英文摘要:
Large Language Models (LLMs) have demonstrated remarkable versatility across various domains. To further advance LLMs, we propose 'SELF' (Self-Evolution with Language Feedback), a novel approach that enables LLMs to self-improve through self-reflection, akin to human learning processes. SELF initiates with a meta-skill learning process that equips the LLMs with capabilities for self-feedback and self-refinement. Subsequently, the model undergoes an iterative process of self-evolution. In each iteration, it utilizes an unlabeled dataset of instructions to generate initial responses. These responses are enhanced through self-feedback and self-refinement. The model is then fine-tuned using this enhanced data. The model undergoes progressive improvement through this iterative self-evolution process. Moreover, the SELF framework enables the model to apply self-refinement during inference, which further improves response quality. Our experiments in mathematics and general tasks demonstrate that SELF can enhance the capabilities of LLMs without human intervention. The SELF framework indicates a promising direction for the autonomous evolution of LLMs, transitioning them from passive information receivers to active participants in their development.
Self-Refine: Iterative Refinement with Self-Feedback
Carnegie Mellon University(卡内基梅隆大学);Allen Institute for Artificial Intelligence(艾伦人工智能研究所);University of Washington(华盛顿大学);NVIDIA(英伟达);UC San Diego(加州大学圣地亚哥分校);Google Research(谷歌研究院)
Aligning Large Language Models from Self-Reference AI Feedback with one General Principle
School of Computer Science, Fudan University(复旦大学计算机科学学院);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);The University of Sydney(悉尼大学);Nanyang Technological University(南洋理工大学)
提出自参考 AI 反馈框架,让 Llama2-Chat 依据通用原则、自我批评与自一致性生成高质量偏好数据,显著提升强化学习对齐效果。
Towards Feedback-to-Plan Decisions for Self-Evolving LLM Agents in CUDA Kernel Generation
Department of Computer Science and Technology, Tsinghua University, Beijing, China(清华大学计算机科学与技术系);Beijing National Research Center for Information Science and Technology, Beijing, China(北京信息科学国家研究中心)
On-Policy Self-Alignment with Fine-grained Knowledge Feedback for Hallucination Mitigation
Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences, Beijing, China(中国科学院软件研究所信息处理实验室,北京,中国);University of Chinese Academy of Sciences, Beijing, China(中国科学院大学,北京,中国);Xiaohongshu Inc(小红书公司)
Exploring the effectiveness of ChatGPT-based feedback compared with teacher feedback and self-feedback: Evidence from Chinese to English translation
School of Foreign Languages, Southeast University(东南大学外国语学院);The Hong Kong Polytechnic University(香港理工大学);School of Languages, Literature, Cultures and Linguistics, Monash University(莫纳什大学语言、文学、文化与语言学学院)