When we design and deploy an Reinforcement Learning (RL) agent, reward functions motivates agents to achieve an objective. An incorrect or incomplete specification of the objective can result in behavior that does not align with human values - failing to adhere with social and moral norms that are ambiguous and context dependent, and cause undesired outcomes such as negative side effects and exploration that is unsafe. Previous work have manually defined reward functions to avoid negative side effects, use human oversight for safe exploration, or use foundation models as planning tools. This work studies the ability of leveraging Large Language Models (LLM)' understanding of morality and social norms on safe exploration augmented RL methods. This work evaluates language model's result against human feedbacks and demonstrates language model's capability as direct reward signals.
Are LLMs Socially Adaptive? Contrasting Belief Evolution in Large Language Models and Humans
Tsinghua University(清华大学);Department of Psychological and Cognitive Sciences(心理与认知科学系);College AI(人工智能学院);School of Management(管理学院);Fudan University(复旦大学);Stevens Institute of Technology(史蒂文斯理工学院);Northwestern University(西北大学);University of Oxford(牛津大学)
Harnessing Natural Fluctuations: Analogue Computer for Efficient Socially Maximal Decision Making
National Institute of Information and Communications Technology(信息通信研究机构);Tokyo Institute of Technology(东京工业大学);Japan Science and Technology Agency(日本科学技术振兴机构);National Institute for Materials Science(物质·材料研究机构)
Aligning to Social Norms and Values in Interactive Narratives
Allen Institute for Artificial Intelligence(艾伦人工智能研究所);University of Washington(华盛顿大学)
我们专注于创建代理,在互动叙事或基于文本的游戏中与社会有益的规范和价值观保持一致-环境中,代理通过自然语言感知并与世界互动。这种交互式代理通常通过强化学习进行训练,以优化任务性能,即使这种奖励可能导致违反社会规范的代理行为-对代理本身或环境中的其他实体造成伤害。社会价值对齐是指创造行为符合特定背景和人群预期道德和社会规范的代理人-在我们的情况下,它意味着代理人的行为方式对自己和他人的危害较小,更有益。我们建立在Jiminy Cricket基准(Hendrycks et al. 2021)的基础上,这是一组25个注释的交互式叙述,包含数千个道德上突出的场景,涵盖了从盗窃和身体伤害到利他主义的一切。我们引入GALAD(游戏-价值对齐通过行动蒸馏)代理,它使用专门训练的语言模型中存在的社会常识知识,根据上下文将其动作空间限制为仅与社会有益价值对齐的那些动作。一项实验研究表明,GALAD代理有效地做出决策,足以将最先进的任务性能提高4%,同时与强大的当代价值对齐方法相比,将社会有害行为的频率降低25%。
Human-Aware Robot Navigation via Reinforcement Learning with Hindsight Experience Replay and Curriculum Learning
The Chinese University of Hong Kong(香港中文大学);Southern University of Science and Technology(南方科技大学);Shenzhen Research Institute of the Chinese University of Hong Kong(香港中文大学深圳研究院)