一种用于反事实公平强化学习的分布映射方法
A Distribution Mapping Approach to Counterfactually Fair Reinforcement Learning
浏览论文内容
中文总结 AI 辅助
该研究针对强化学习在高风险场景中可能出现的不公平决策问题，提出一种结合策略学习的反事实公平数据预处理算法，通过分位数分布映射方法实现反事实状态与奖励估计，并在数值实验和真实健康数据集上验证了算法有效性。
中文摘要 AI 辅助
强化学习（RL）旨在优化序列决策以随时间最大化总体层面的收益。然而，当RL被部署在医疗保健等高风险场景中时，其决策可能会系统性地限制某些亚种群获取有价值服务的机会，这与利益相关者的价值观和目标相悖。反事实公平（CF）提供了一个基于因果推理解决该问题的有前景的框架。本文开发了一种数据预处理算法，当与策略学习结合使用时，可使RL实现反事实公平。该算法依赖于一种新颖的分位数分布映射方法，用于在数据预处理步骤中依次估计反事实状态和奖励，将反事实预测常用的可加性假设作为特例包含在内。我们从理论上证明，在温和的正则性条件下，每一步的反事实不公平程度和无限时域的次优性差距都可以被界定。我们还在数值实验以及对真实世界介入式数字健康数据集的应用中对该算法进行了实证测试。
英文摘要
Reinforcement learning (RL) seeks to optimize sequential decisions to maximize population-level benefits over time. However, when deployed in high-stakes settings such as healthcare, RL decisions might systematically restrict some subpopulation's access to valuable services in a manner contrary to the values and goals of stakeholders. Counterfactual fairness (CF) offers a promising framework to address this problem based on causal reasoning. This paper develops a data preprocessing algorithm that, when used in tandem with policy learning, enables CF in RL. Our algorithm relies on a novel quantile distribution mapping method for sequentially estimating the counterfactual states and rewards in the data preprocessing step, subsuming common additivity assumptions used for counterfactual prediction as a special case. We theoretically prove that the per-step level of counterfactual unfairness and infinite-horizon suboptimality gap can be bounded under mild regularity conditions. We also empirically test our algorithm in numerical experiments as well as in application to a real-world interventional digital health dataset.
发表机构
- University of Michigan（密歇根大学）
- LSE（伦敦政治经济学院）
机构由 AI 辅助整理，请以论文原文为准。
