通过比较反馈获取用户偏好以实现个性化多目标决策
Eliciting User Preferences for Personalized Multi-Objective Decision Making through Comparative Feedback
AI总结：
本文提出一种多目标决策框架，通过策略比较或代表性轨迹比较学习用户偏好向量，并以少量比较查询高效计算个性化近最优策略。
摘要：
在经典强化学习（RL）和决策问题中，策略通常依据标量奖励函数进行评估，所有最优策略在期望回报方面是相同的。然而，许多现实世界问题涉及平衡多个、有时相互冲突的目标，而这些目标的相对优先级会因每个用户的偏好而有所不同。因此，对一个用户最优的策略对另一个用户可能是次优的。在这项工作中，我们提出了一个多目标决策框架，该框架能够适应不同用户在目标上的偏好，其中偏好通过策略比较来学习。我们的模型由一个具有向量值奖励函数的马尔可夫决策过程组成，每个用户都有一个未知的偏好向量，用于表达每个目标的相对重要性。目标是高效地为给定用户计算一个近最优策略。我们考虑了两种用户反馈模型。我们首先处理的情况是，向用户提供两个策略，并返回他们偏好的策略作为反馈。然后，我们转向另一种用户反馈模型，在该模型中，向用户提供两组小型的加权代表性轨迹集合，并选择其偏好的那一组。在这两种情况下，我们都提出了一种算法，该算法使用少量比较查询即可为用户找到近最优策略。
英文摘要：
In classic reinforcement learning (RL) and decision making problems, policies are evaluated with respect to a scalar reward function, and all optimal policies are the same with regards to their expected return. However, many real-world problems involve balancing multiple, sometimes conflicting, objectives whose relative priority will vary according to the preferences of each user. Consequently, a policy that is optimal for one user might be sub-optimal for another. In this work, we propose a multi-objective decision making framework that accommodates different user preferences over objectives, where preferences are learned via policy comparisons. Our model consists of a Markov decision process with a vector-valued reward function, with each user having an unknown preference vector that expresses the relative importance of each objective. The goal is to efficiently compute a near-optimal policy for a given user. We consider two user feedback models. We first address the case where a user is provided with two policies and returns their preferred policy as feedback. We then move to a different user feedback model, where a user is instead provided with two small weighted sets of representative trajectories and selects the preferred one. In both cases, we suggest an algorithm that finds a nearly optimal policy for the user using a small number of comparison queries.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
