arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.20728cs.AI
收藏

在人工智能反馈中平衡城市交通控制的多个目标

Balancing Multiple Objectives in Urban Traffic Control with Reinforcement Learning from AI Feedback

Trinity College Dublin(都柏林三一学院)

Chenyang Zhao, Vinny Cahill, Ivana Dusparic

AI总结:

本文提出利用人工智能反馈方法在城市交通控制中平衡多个目标,通过多目标自适应系统实现无需繁琐奖励工程的策略学习。

摘要:

奖励设计一直是现实世界强化学习(RL)部署中的核心挑战,尤其是在涉及多个目标的设置中。基于偏好的强化学习提供了一种替代方法,通过学习人类对行为结果的偏好来学习。最近,从人工智能反馈(RLAIF)表明,大型语言模型(LLMs)可以生成偏好标签,从而减少对人工标注者的依赖。然而,现有RLAIF工作通常只关注单目标任务,留下了开放性问题:RLAIF如何处理涉及多个目标的系统。在这样的系统中,冲突目标之间的权衡难以指定,策略有风险会陷入只优化主导目标。在本文中,我们探讨了将RLAIF范式扩展到多目标自适应系统。我们证明多目标RLAIF可以产生产生平衡权衡的策略,反映不同的用户优先级,而无需繁琐的奖励工程。我们认为,将RLAIF整合到多目标RL中,为在本质上存在冲突目标的领域提供了一条可扩展的路径,以实现用户对齐的策略学习。

英文摘要:

Reward design has been one of the central challenges for real world reinforcement learning (RL) deployment, especially in settings with multiple objectives. Preference-based RL offers an appealing alternative by learning from human preferences over pairs of behavioural outcomes. More recently, RL from AI feedback (RLAIF) has demonstrated that large language models (LLMs) can generate preference labels at scale, mitigating the reliance on human annotators. However, existing RLAIF work typically focuses only on single-objective tasks, leaving the open question of how RLAIF handles systems that involve multiple objectives. In such systems trade-offs among conflicting objectives are difficult to specify, and policies risk collapsing into optimizing for a dominant goal. In this paper, we explore the extension of the RLAIF paradigm to multi-objective self-adaptive systems. We show that multi-objective RLAIF can produce policies that yield balanced trade-offs reflecting different user priorities without laborious reward engineering. We argue that integrating RLAIF into multi-objective RL offers a scalable path toward user-aligned policy learning in domains with inherently conflicting objectives.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑