arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2405.13022cs.CLcs.LG
收藏

LLMs 可以通过迭代自反思学习自我克制

LLMs can learn self-restraint through iterative self-reflection

ServiceNow Research(ServiceNow研究院);Mila, Université de Montréal(蒙特利尔大学Mila研究所);Mila, McGill University(麦吉尔大学Mila研究所);Mila, Polytechnique Montréal(蒙特利尔理工学院Mila研究所)

Alexandre Piché, Aristides Milios, Dzmitry Bahdanau, Chris Pal

AI总结:

本文提出 ReSearch 自反思算法和效用函数,通过迭代自提示与自评估生成合成数据微调 LLMs,使其学会在不确定时选择性克制或弃权,从而在不增加推理成本的情况下减少幻觉。

摘要:

为了安全部署,大型语言模型(LLMs)必须具备根据自身知识水平以及与特定主题相关的不确定性动态调整行为的能力。我们将这种自适应行为称为自我克制,由于它依赖于 LLM 的内部知识,因此教授这种能力并非易事。默认情况下,LLMs 被训练为最大化下一个 token 的似然,这并不会教会模型根据其不确定性水平来调节回答。为了学习自我克制,我们设计了一个效用函数,可以鼓励模型仅在对其回答有信心时才产生响应。该效用函数可用于对不同长度和弃权(不执行)的生成进行评分。为了优化该函数,我们引入了 ReSearch,这是一个由迭代自提示和自我评估组成的“自反思”过程。我们使用 ReSearch 算法生成合成数据,并在这些数据上微调我们的模型。与原始版本相比,我们得到的模型在已知和未知主题上整体产生更少的幻觉,且不增加额外的推理成本,因为模型学会了选择性地克制自己。此外,我们的方法通过在搜索过程中用表达弃权(不执行)的回答来扩充模型生成的样本,优雅地融入了弃权能力。

英文摘要:

In order to be deployed safely, Large Language Models (LLMs) must be capable of dynamically adapting their behavior based on their level of knowledge and uncertainty associated with specific topics. This adaptive behavior, which we refer to as self-restraint, is non-trivial to teach since it depends on the internal knowledge of an LLM. By default, LLMs are trained to maximize the next token likelihood, which does not teach the model to modulate its answer based on its level of uncertainty. In order to learn self-restraint, we devise a utility function that can encourage the model to produce responses only when it is confident in them. This utility function can be used to score generation of different length and abstention. To optimize this function, we introduce ReSearch, a process of "self-reflection" consisting of iterative self-prompting and self-evaluation. We use the ReSearch algorithm to generate synthetic data on which we finetune our models. Compared to their original versions, our resulting models generate fewer \emph{hallucinations} overall at no additional inference cost, for both known and unknown topics, as the model learns to selectively restrain itself. In addition, our method elegantly incorporates the ability to abstain by augmenting the samples generated by the model during the search procedure with an answer expressing abstention.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑