面向大语言模型的主动偏好学习
Active Preference Learning for Large Language Models
AI总结：
本文针对DPO提出一种基于预测熵与隐式偏好模型确定性的主动学习采集函数，以更高效利用偏好标签，提升大语言模型微调的学习速率与最终性能。
摘要：
随着大语言模型（LLMs）能力不断增强，用于与人类意图对齐的微调技术变得日益重要。对齐这些模型的一个关键考虑因素是如何最有效地利用人力资源，或在将LLMs本身用作预言机（oracle）的情况下利用模型资源。基于人类或AI偏好的强化学习（RLHF/RLAIF）是此类技术中最突出的例子，但该方法复杂且常常不稳定。最近提出的直接偏好优化（DPO）是一种更简单且更稳定的替代方案。在本工作中，我们为DPO开发了一种主动学习策略，以更好地利用偏好标签。我们基于语言模型的预测熵以及由DPO优化的隐式偏好模型的确定性度量，提出了一种实用的针对提示/补全对的采集函数。我们展示了我们的方法如何同时提高在成对偏好数据上进行微调的学习速率和最终性能。
英文摘要：
As large language models (LLMs) become more capable, fine-tuning techniques for aligning with human intent are increasingly important. A key consideration for aligning these models is how to most effectively use human resources, or model resources in the case where LLMs themselves are used as oracles. Reinforcement learning from Human or AI preferences (RLHF/RLAIF) is the most prominent example of such a technique, but is complex and often unstable. Direct Preference Optimization (DPO) has recently been proposed as a simpler and more stable alternative. In this work, we develop an active learning strategy for DPO to make better use of preference labels. We propose a practical acquisition function for prompt/completion pairs based on the predictive entropy of the language model and a measure of certainty of the implicit preference model optimized by DPO. We demonstrate how our approach improves both the rate of learning and final performance of fine-tuning on pairwise preference data.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
