arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2501.17112cs.LG
收藏

解码对齐中的人类偏好:一种改进的逆宪法AI方法

Decoding Human Preferences in Alignment: An Improved Approach to Inverse Constitutional AI

Cornell University(康奈尔大学)

Carl-Leander Henneking, Claas Beger

AI总结:

本文改进了逆宪法AI算法,通过优化原则生成、聚类和嵌入过程,从偏好数据中提取更准确、泛化的显式原则,以增强LLM对齐的可解释性和适应性。

摘要:

传统的大语言模型(LLM)对齐方法,如基于人类反馈的强化学习(RLHF)和直接偏好优化(DPO),依赖于隐含原则,限制了可解释性。宪法AI(CAI)提供了一种显式的、基于规则的框架来指导LLM对齐。在此基础上,我们改进了逆宪法AI(ICAI)算法,该算法从偏好数据集中提取宪法。通过改进原则生成、聚类和嵌入过程,我们的方法增强了在合成和真实世界数据集中提取原则的准确性和泛化性。我们的结果突显了这些原则在促进更透明和适应性更强的对齐方法方面的潜力,为超越传统微调的未来进展提供了一个有前景的方向。

英文摘要:

Traditional methods for aligning Large Language Models (LLMs), such as Reinforcement Learning from Human Feedback (RLHF) and Direct Preference Optimization (DPO), rely on implicit principles, limiting interpretability. Constitutional AI (CAI) offers an explicit, rule-based framework for guiding LLM alignment. Building on this, we refine the Inverse Constitutional AI (ICAI) algorithm, which extracts constitutions from preference datasets. By improving principle generation, clustering, and embedding processes, our approach enhances the accuracy and generalizability of extracted principles across synthetic and real-world datasets. Our results highlight the potential of these principles to foster more transparent and adaptable alignment methods, offering a promising direction for future advancements beyond traditional fine-tuning.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑