arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2310.00533cs.CLcs.AIcs.LG
收藏

SELF:基于语言反馈的自我进化

SELF: Self-Evolution with Language Feedback

The University of Hong Kong(香港大学)

Jianqiao Lu, Wanjun Zhong, Wenyong Huang, Yufei Wang, Qi Zhu, Fei Mi, Baojun Wang, Weichao Wang, Xingshan Zeng, Lifeng Shang, Xin Jiang, Qun Liu

AI总结:

本研究提出SELF框架,通过元技能学习让大语言模型具备自我反馈与优化能力,经迭代自我进化及推理阶段自我优化,无需人工干预即可提升其在数学和通用任务上的表现,为LLM自主进化提供新方向。

摘要:

大语言模型(LLMs)已在诸多领域展现出卓越的通用性。为进一步推动LLMs发展,我们提出了SELF(Self-Evolution with Language Feedback,基于语言反馈的自我进化)这一全新方法,它能让LLMs像人类学习过程一样通过自我反思实现自我提升。SELF首先启动元技能学习流程,让LLMs具备自我反馈和自我优化的能力。随后,模型进入迭代式自我进化阶段:每一轮迭代中,模型利用无标注指令数据集生成初始响应,再通过自我反馈和自我优化对这些响应进行增强,之后用增强后的数据对模型进行微调,模型通过这一迭代自我进化过程实现逐步提升。此外,SELF框架支持模型在推理阶段应用自我优化,进一步提升响应质量。我们在数学任务和通用任务上开展的实验表明,SELF无需人工干预即可增强LLMs的能力。SELF框架为LLMs的自主进化指明了极具前景的方向,推动其从被动的信息接收者转变为自身发展过程中的主动参与者。

英文摘要:

Large Language Models (LLMs) have demonstrated remarkable versatility across various domains. To further advance LLMs, we propose 'SELF' (Self-Evolution with Language Feedback), a novel approach that enables LLMs to self-improve through self-reflection, akin to human learning processes. SELF initiates with a meta-skill learning process that equips the LLMs with capabilities for self-feedback and self-refinement. Subsequently, the model undergoes an iterative process of self-evolution. In each iteration, it utilizes an unlabeled dataset of instructions to generate initial responses. These responses are enhanced through self-feedback and self-refinement. The model is then fine-tuned using this enhanced data. The model undergoes progressive improvement through this iterative self-evolution process. Moreover, the SELF framework enables the model to apply self-refinement during inference, which further improves response quality. Our experiments in mathematics and general tasks demonstrate that SELF can enhance the capabilities of LLMs without human intervention. The SELF framework indicates a promising direction for the autonomous evolution of LLMs, transitioning them from passive information receivers to active participants in their development.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑