ICDPO:通过上下文直接偏好优化有效借用他人的对齐能力
ICDPO: Effectively Borrowing Alignment Capability of Others via In-context Direct Preference Optimization
AI总结:
本文提出ICDPO方法,通过上下文学习前后LLM状态构建即时评分器,使LLM能从更优模型借用人类偏好对齐能力,生成对齐良好的响应,性能优于免微调基线且媲美SFT+LoRA。
摘要:
大型语言模型(LLMs)依赖人类偏好对齐(HPA)来确保生成安全内容。由于微调相关成本高昂,免微调方法应运而生,通常通过外部辅助方法修改LLM解码。然而,这些方法并未从根本上增强LLM本身。本文重新思考DPO的推导过程,据此利用上下文学习(ICL)前后LLM的状态反向构建即时评分器。相应地,我们提出一种名为上下文直接偏好优化(ICDPO)的新方法。它使LLM能够通过ICL从更优的LLM借用HPA能力,生成如上述即时评分器所估计的对齐良好的响应,从而提升最终性能。ICDPO可通过两阶段检索器和升级版评分器进一步增强,两者均提供益处。大量实验表明其有效性,特别是在优于两个免微调基线方面,并且展现出与SFT + LoRA相媲美的竞争力。我们还进行了详细分析,为ICDPO提供全面见解。
英文摘要:
Large Language Models (LLMs) rely on Human Preference Alignment (HPA) to ensure the generation of safe content. Due to the heavy cost associated with fine-tuning, fine-tuning-free methods have emerged, typically modifying LLM decoding with external auxiliary methods. However, these methods do not essentially enhance the LLM itself. In this paper, we rethink the derivation procedures of DPO, based on which we conversely build an instant scorer using the states of the LLM before and after In-context Learning (ICL). Accordingly, we propose a novel approach called In-Context Direct Preference Optimization (ICDPO). It enables LLMs to borrow the HPA capabilities from superior LLMs with ICL, generating well-aligned responses as estimated by the aforementioned instant scorer, thereby enhancing the final performance. ICDPO can be further enhanced with a two-stage retriever and an upgraded scorer, both offering benefits. Extensive experiments show its effectiveness, particularly in outperforming two fine-tuning-free baselines, and it exhibits competitiveness with SFT + LoRA. We also conduct detailed analyses to offer comprehensive insights into ICDPO.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。