arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2401.12459cs.AI
收藏

迈向具有社会与道德意识的RL智能体:基于LLM的奖励设计

Towards Socially and Morally Aware RL agent: Reward Design With LLM

Zhaoyue Wang

AI总结:

本研究针对RL智能体目标设定不完整导致违背社会道德规范的问题,提出利用LLM对道德与社会规范的理解作为安全探索增强RL方法的直接奖励信号,并验证了其有效性。

摘要:

在设计和部署强化学习(RL)智能体时,奖励函数会激励智能体实现目标。目标设定错误或不完整可能导致行为与人类价值观不符——无法遵守模糊且依赖语境的社会与道德规范,并导致负面副作用和不安全探索等不良后果。以往研究通过手动定义奖励函数来避免负面副作用,利用人类监督实现安全探索,或将基础模型作为规划工具。本研究探讨了利用大型语言模型(LLM)对道德和社会规范的理解能力,应用于安全探索增强的RL方法。本研究将语言模型的结果与人类反馈进行评估对比,并展示了语言模型作为直接奖励信号的能力。

英文摘要:

When we design and deploy an Reinforcement Learning (RL) agent, reward functions motivates agents to achieve an objective. An incorrect or incomplete specification of the objective can result in behavior that does not align with human values - failing to adhere with social and moral norms that are ambiguous and context dependent, and cause undesired outcomes such as negative side effects and exploration that is unsafe. Previous work have manually defined reward functions to avoid negative side effects, use human oversight for safe exploration, or use foundation models as planning tools. This work studies the ability of leveraging Large Language Models (LLM)' understanding of morality and social norms on safe exploration augmented RL methods. This work evaluates language model's result against human feedbacks and demonstrates language model's capability as direct reward signals.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑