通过不确定性增强偏好优化实现自进化大语言模型
Self-Evolutionary Large Language Models through Uncertainty-Enhanced Preference Optimization
AI总结:
针对迭代偏好优化中噪声偏好数据过多的问题,提出结合BNN与MC dropout的UPO框架及自进化算法,通过成对不确定性估计与可靠反馈采样缓解噪声,提升LLM性能。
摘要:
迭代偏好优化近来已成为大语言模型(LLM)的事实训练范式之一,但由于循环中产生的噪声偏好数据过多,其性能仍不尽如人意。为解决这一问题,我们提出了不确定性增强偏好优化(Uncertainty-enhanced Preference Optimization, UPO)框架,使LLM能够借助可靠反馈实现自进化。该框架的核心思路是通过执行成对不确定性估计与审慎的可靠反馈采样,减少来自当前策略和奖励模型的噪声偏好数据。为此,我们引入了一个估计器模型,该模型在贝叶斯神经网络(Bayesian neural network, BNN)中结合蒙特卡洛(Monte Carlo, MC)dropout,对来自LLM策略的偏好数据进行不确定性估计。与现有直接基于奖励分数过滤生成响应的方法相比,该估计器以成对方式关注模型不确定性,有效规避了奖励模型的确认偏差问题。此外,我们还提出了一种不确定性增强自进化算法,以提升偏好优化的鲁棒性,促使LLM生成兼具高奖励与高确定性的响应。在多个基准上开展的大量实验表明,我们的框架显著缓解了噪声问题,提升了迭代偏好优化的性能。
英文摘要:
Iterative preference optimization has recently become one of the de-facto training paradigms for large language models (LLMs), but the performance is still underwhelming due to too much noisy preference data yielded in the loop. To combat this issue, we present an \textbf{U}ncertainty-enhanced \textbf{P}reference \textbf{O}ptimization (UPO) framework to make the LLM self-evolve with reliable feedback. The key idea is mitigating the noisy preference data derived from the current policy and reward models by performing pair-wise uncertainty estimation and judiciously reliable feedback sampling. To reach this goal, we thus introduce an estimator model, which incorporates Monte Carlo (MC) dropout in Bayesian neural network (BNN) to perform uncertainty estimation for the preference data derived from the LLM policy. Compared to the existing methods that directly filter generated responses based on the reward score, the estimator focuses on the model uncertainty in a pair-wise manner and effectively bypasses the confirmation bias problem of the reward model. Additionally, we also propose an uncertainty-enhanced self-evolution algorithm to improve the robustness of preference optimization and encourage the LLM to generate responses with both high reward and certainty. Extensive experiments over multiple benchmarks demonstrate that our framework substantially alleviates the noisy problem and improves the performance of iterative preference optimization.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。