arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2312.15907cs.CL
收藏

即时对齐:让聊天机器人的行为适应既定规范

Align on the Fly: Adapting Chatbot Behavior to Established Norms

Shanghai Jiao Tong University(上海交通大学);The Hong Kong Polytechnic University(香港理工大学);Generative AI Research Lab (GAIR)(生成式人工智能研究实验室(GAIR));Carnegie Mellon University(卡内基梅隆大学);University of Waterloo(滑铁卢大学);The Hong Kong University of Science and Technology(香港科技大学);Google DeepMind(谷歌DeepMind);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室)

Chunpu Xu, Steffi Chern, Ethan Chern, Ge Zhang, Zekun Wang, Ruibo Liu, Jing Li, Jie Fu, Pengfei Liu

AI总结:

针对大语言模型与动态人类价值观对齐的挑战,提出即时偏好优化(OPO)方法,利用外部记忆存储规则实现无需训练的实时对齐,并在法律与道德领域验证了其有效性。

摘要:

在本文中,我们旨在使大语言模型与随时间与地点不断变化、复杂且多样的人类价值观(如社会规范)对齐。这对现有的对齐技术(如监督微调)构成了挑战,因为这些技术将价值观内化于模型参数中。为克服这一挑战,我们提出了一种即时偏好优化(OPO)方法,这是一种以流式方式工作的实时对齐方法。它利用外部记忆存储用于对齐的既定规则,无需进一步训练即可约束大语言模型的行为,从而便于便捷地更新和定制人类价值观。我们还引入了一种可扩展的评估方法,以更有效地评估所提出的方法。在来自法律和道德领域的人工标注和自动生成问题上的实验结果表明,所提出的OPO方法具有有效性。我们的代码和数据已发布在https://github.com/GAIR-NLP/OPO。

英文摘要:

In this paper, we aim to align large language models with the ever-changing, complex, and diverse human values (e.g., social norms) across time and locations. This presents a challenge to existing alignment techniques, such as supervised fine-tuning, which internalize values within model parameters. To overcome this, we propose an On-the-fly Preference Optimization (OPO) method, which is a real-time alignment that works in a streaming way. It employs an external memory to store established rules for alignment, which can constrain LLMs' behaviors without further training, allowing for convenient updates and customization of human values. We also introduce a scalable evaluation to assess the proposed method more effectively. Experimental results on both human-annotated and auto-generated questions from legal and moral domains indicate the effectiveness of the proposed OPO method. Our code and data are released at https://github.com/GAIR-NLP/OPO.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑