本文提出 ReSearch 自反思算法和效用函数,通过迭代自提示与自评估生成合成数据微调 LLMs,使其学会在不确定时选择性克制或弃权,从而在不增加推理成本的情况下减少幻觉。
摘要:
为了安全部署,大型语言模型(LLMs)必须具备根据自身知识水平以及与特定主题相关的不确定性动态调整行为的能力。我们将这种自适应行为称为自我克制,由于它依赖于 LLM 的内部知识,因此教授这种能力并非易事。默认情况下,LLMs 被训练为最大化下一个 token 的似然,这并不会教会模型根据其不确定性水平来调节回答。为了学习自我克制,我们设计了一个效用函数,可以鼓励模型仅在对其回答有信心时才产生响应。该效用函数可用于对不同长度和弃权(不执行)的生成进行评分。为了优化该函数,我们引入了 ReSearch,这是一个由迭代自提示和自我评估组成的“自反思”过程。我们使用 ReSearch 算法生成合成数据,并在这些数据上微调我们的模型。与原始版本相比,我们得到的模型在已知和未知主题上整体产生更少的幻觉,且不增加额外的推理成本,因为模型学会了选择性地克制自己。此外,我们的方法通过在搜索过程中用表达弃权(不执行)的回答来扩充模型生成的样本,优雅地融入了弃权能力。
英文摘要:
In order to be deployed safely, Large Language Models (LLMs) must be capable of dynamically adapting their behavior based on their level of knowledge and uncertainty associated with specific topics. This adaptive behavior, which we refer to as self-restraint, is non-trivial to teach since it depends on the internal knowledge of an LLM. By default, LLMs are trained to maximize the next token likelihood, which does not teach the model to modulate its answer based on its level of uncertainty. In order to learn self-restraint, we devise a utility function that can encourage the model to produce responses only when it is confident in them. This utility function can be used to score generation of different length and abstention. To optimize this function, we introduce ReSearch, a process of "self-reflection" consisting of iterative self-prompting and self-evaluation. We use the ReSearch algorithm to generate synthetic data on which we finetune our models. Compared to their original versions, our resulting models generate fewer \emph{hallucinations} overall at no additional inference cost, for both known and unknown topics, as the model learns to selectively restrain itself. In addition, our method elegantly incorporates the ability to abstain by augmenting the samples generated by the model during the search procedure with an answer expressing abstention.
Instruct-of-Reflection: Enhancing Large Language Models Iterative Reflection Capabilities via Dynamic-Meta Instruction
Beijing University of Posts and Telecommunications(北京邮电大学);AI Lab of Lenovo Research(联想研究院AI实验室);Tianjin University(天津大学);The Hong Kong University of Science and Technology(香港科技大学)
Advancing Multi-Step Mathematical Reasoning in Large Language Models through Multi-Layered Self-Reflection with Auto-Prompting
Luiz de Queiroz College of Agriculture University of São Paulo(圣保罗大学农业学院);School of Engineering and Sciences Tecnologico de Monterrey(蒙特雷技术学院工程与科学学院);Institute of Mathematics and Computer Sciences University of São Paulo(圣保罗大学数学与计算机科学研究所)
A Training-Free Regeneration Paradigm: Contrastive Reflection Memory Guided Self-Verification and Self-Improvement
Department of Electrical and Computer Engineering, McGill University(麦吉尔大学电气与计算机工程系);Department of Mechanical, Industrial and Aerospace Engineering, Concordia University(Concordia大学机械、工业与航空航天工程系)
Towards Stepwise Domain Knowledge-Driven Reasoning Optimization and Reflection Improvement
College of Computer Science and Technology, Zhejiang University(浙江大学计算机科学与技术学院);Tongyi Lab, Alibaba Group(阿里巴巴集团通义实验室);Law&AI Lab, Zhejiang University(浙江大学法律与人工智能实验室)
Agent-Pro: Learning to Evolve via Policy-Level Reflection and Optimization
College of Computer Science and Technology, Zhejiang University(浙江大学计算机科学与技术学院);Institute of Software, Chinese Academy of Sciences(中国科学院软件研究所);Nanjing Institute of Software Technology(南京软件技术学院);Nanjing University of Posts and Telecommunications(南京邮电大学);University of Chinese Academy of Sciences(中国科学院大学);Beijing University of Technology(北京工业大学);Nanjing University of Information Science and Technology(南京信息工程大学)
ReBeCA: Causality-Inspired Behavior Analysis for Understanding the Self-Reflection of Language Models
Monash University(莫纳什大学);Harbin Institute of Technology(哈尔滨工业大学);Hainan University(海南大学);South China University of Technology(华南理工大学);The Chinese University of Hong Kong(香港中文大学)