即时对齐:让聊天机器人的行为适应既定规范
Align on the Fly: Adapting Chatbot Behavior to Established Norms
AI总结:
针对大语言模型与动态人类价值观对齐的挑战,提出即时偏好优化(OPO)方法,利用外部记忆存储规则实现无需训练的实时对齐,并在法律与道德领域验证了其有效性。
摘要:
在本文中,我们旨在使大语言模型与随时间与地点不断变化、复杂且多样的人类价值观(如社会规范)对齐。这对现有的对齐技术(如监督微调)构成了挑战,因为这些技术将价值观内化于模型参数中。为克服这一挑战,我们提出了一种即时偏好优化(OPO)方法,这是一种以流式方式工作的实时对齐方法。它利用外部记忆存储用于对齐的既定规则,无需进一步训练即可约束大语言模型的行为,从而便于便捷地更新和定制人类价值观。我们还引入了一种可扩展的评估方法,以更有效地评估所提出的方法。在来自法律和道德领域的人工标注和自动生成问题上的实验结果表明,所提出的OPO方法具有有效性。我们的代码和数据已发布在https://github.com/GAIR-NLP/OPO。
英文摘要:
In this paper, we aim to align large language models with the ever-changing, complex, and diverse human values (e.g., social norms) across time and locations. This presents a challenge to existing alignment techniques, such as supervised fine-tuning, which internalize values within model parameters. To overcome this, we propose an On-the-fly Preference Optimization (OPO) method, which is a real-time alignment that works in a streaming way. It employs an external memory to store established rules for alignment, which can constrain LLMs' behaviors without further training, allowing for convenient updates and customization of human values. We also introduce a scalable evaluation to assess the proposed method more effectively. Experimental results on both human-annotated and auto-generated questions from legal and moral domains indicate the effectiveness of the proposed OPO method. Our code and data are released at https://github.com/GAIR-NLP/OPO.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。