arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2506.08062cs.LGcs.AI
收藏

FairDICE:公平性驱动的离线多目标强化学习

FairDICE: Fairness-Driven Offline Multi-Objective Reinforcement Learning

Korea University(韩国大学);Yonsei University(延世大学);Gauss Labs Inc.(Gauss实验室公司)

Woosung Kim, Jinho Lee, Jongmin Lee, Byung-Jun Lee

AI总结:

针对离线多目标强化学习中非线性公平福利目标优化的空白,提出FairDICE框架,借助分布校正估计实现无需偏好权重搜索的稳定高效学习,在基准测试中取得优于基线的公平感知性能。

摘要:

多目标强化学习(MORL)旨在优化存在冲突目标场景下的策略,通常采用线性标量化方法将向量形式的回报转化为标量信号。该方法虽对部分偏好场景有效,但无法实现纳什社会福利、最大最小公平这类需要非线性、非加性权衡的公平导向目标。目前已有若干面向特定公平目标的在线算法被提出,但在必须基于固定数据集开展学习的离线场景下,用于优化非线性福利准则的统一方法仍属研究空白。本研究提出首个可直接优化非线性福利目标的离线MORL框架FairDICE,该框架利用分布校正估计同时兼顾福利最大化与分布正则化,无需显式设置偏好权重或开展穷尽权重搜索即可实现稳定、高样本效率的学习。在多个离线基准测试中,FairDICE相较现有基线展现出更优异的公平感知性能。

英文摘要:

Multi-objective reinforcement learning (MORL) aims to optimize policies in the presence of conflicting objectives, where linear scalarization is commonly used to reduce vector-valued returns into scalar signals. While effective for certain preferences, this approach cannot capture fairness-oriented goals such as Nash social welfare or max-min fairness, which require nonlinear and non-additive trade-offs. Although several online algorithms have been proposed for specific fairness objectives, a unified approach for optimizing nonlinear welfare criteria in the offline setting-where learning must proceed from a fixed dataset-remains unexplored. In this work, we present FairDICE, the first offline MORL framework that directly optimizes nonlinear welfare objective. FairDICE leverages distribution correction estimation to jointly account for welfare maximization and distributional regularization, enabling stable and sample-efficient learning without requiring explicit preference weights or exhaustive weight search. Across multiple offline benchmarks, FairDICE demonstrates strong fairness-aware performance compared to existing baselines.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑