Complex real-world applications of cyber-physical systems give rise to the need for multi-objective controller synthesis, which concerns the problem of computing an optimal controller subject to multiple (possibly conflicting) criteria. The relative importance of objectives is often specified by human decision-makers. However, there is inherent uncertainty in human preferences (e.g., due to artifacts resulting from different preference elicitation methods). In this paper, we formalize the notion of uncertain human preferences and present a novel approach that accounts for this uncertainty in the context of multi-objective controller synthesis for Markov decision processes (MDPs). Our approach is based on mixed-integer linear programming and synthesizes an optimally permissive multi-strategy that satisfies uncertain human preferences with respect to a multi-objective property. Experimental results on a range of large case studies show that the proposed approach is feasible and scalable across varying MDP model sizes and uncertainty levels of human preferences. Evaluation via an online user study also demonstrates the quality and benefits of the synthesized controllers.
Radboud University（拉德堡德大学）；University of Oxford（牛津大学）
Capturing uncertainty in models of complex dynamical systems is crucial to designing safe controllers. Stochastic noise causes aleatoric uncertainty, whereas imprecise knowledge of model parameters leads to epistemic uncertainty. Several approaches use formal abstractions to synthesize policies that satisfy temporal specifications related to safety and reachability. However, the underlying models exclusively capture aleatoric but not epistemic uncertainty, and thus require that model parameters are known precisely. Our contribution to overcoming this restriction is a novel abstraction-based co…
Existing observational approaches for learning human preferences, such as inverse reinforcement learning, usually make strong assumptions about the observability of the human's environment. However, in reality, people make many important decisions under uncertainty. To better understand preference learning in these cases, we study the setting of inverse decision theory (IDT), a previously proposed framework where a human is observed making non-sequential binary decisions under uncertainty. In IDT, the human's preferences are conveyed through their loss function, which expresses a tradeoff betw…
Conveying complex objectives to reinforcement learning (RL) agents often requires meticulous reward engineering. Preference-based RL methods are able to learn a more flexible reward model based on human preferences by actively incorporating human feedback, i.e. teacher's preferences between two clips of behaviors. However, poor feedback-efficiency still remains a problem in current preference-based RL algorithms, as tailored human feedback is very expensive. To handle this issue, previous methods have mainly focused on improving query selection and policy initialization. At the same time, rece…
While deep reinforcement learning (RL) agents have demonstrated incredible potential in attaining dexterous behaviours for robotics, they tend to make errors when deployed in the real world due to mismatches between the training and execution environments. In contrast, the classical robotics community have developed a range of controllers that can safely operate across most states in the real world given their explicit derivation. These controllers however lack the dexterity required for complex tasks given limitations in analytical modelling and approximations. In this paper, we propose Bayes…
University of Texas at Austin（德克萨斯大学奥斯汀分校）；Radboud University（拉德堡德大学）；University of California, Berkeley（加州大学伯克利分校）；Eindhoven University of Technology（埃因霍温理工大学）
Uncertain partially observable Markov decision processes (uPOMDPs) allow the probabilistic transition and observation functions of standard POMDPs to belong to a so-called uncertainty set. Such uncertainty, referred to as epistemic uncertainty, captures uncountable sets of probability distributions caused by, for instance, a lack of data available. We develop an algorithm to compute finite-memory policies for uPOMDPs that robustly satisfy specifications against any admissible distribution. In general, computing such policies is theoretically and practically intractable. We provide an efficient…
Albert-Ludwigs-Universität Freiburg（阿尔伯特-路德维希-弗莱堡大学）；Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）；Bosch Center for Artificial Intelligence (BCAI)（博世人工智能中心（BCAI））
While classic control theory offers state of the art solutions in many problem scenarios, it is often desired to improve beyond the structure of such solutions and surpass their limitations. To this end, residual policy learning (RPL) offers a formulation to improve existing controllers with reinforcement learning (RL) by learning an additive "residual" to the output of a given controller. However, the applicability of such an approach highly depends on the structure of the controller. Often, internal feedback signals of the controller limit an RL algorithm to adequately change the policy and,…
Facing an unknown situation, a person may not be able to firmly elicit his/her preferences over different alternatives, so he/she tends to express uncertain preferences. Given a community of different persons expressing their preferences over certain alternatives under uncertainty, to get a collective representative opinion of the whole community, a preference fusion process is required. The aim of this work is to propose a preference fusion method that copes with uncertainty and escape from the Condorcet paradox. To model preferences under uncertainty, we propose to develop a model of prefere…
In this paper we propose efficient methods for elicitation of complexly structured preferences and utilize these in problems of decision making under (severe) uncertainty. Based on the general framework introduced in Jansen, Schollmeyer and Augustin (2018, Int. J. Approx. Reason), we now design elicitation procedures and algorithms that enable decision makers to reveal their underlying preference system (i.e. two relations, one encoding the ordinal, the other the cardinal part of the preferences) while having to answer as few as possible simple ranking questions. Here, two different approaches…
University of Bonn（波恩大学）；Fraunhofer FKIE（弗劳恩霍夫通信、信息处理与人机工程研究所）
For the most comfortable, human-aware robot navigation, subjective user preferences need to be taken into account. This paper presents a novel reinforcement learning framework to train a personalized navigation controller along with an intuitive virtual reality demonstration interface. The conducted user study provides evidence that our personalized approach significantly outperforms classical approaches with more comfortable human-robot experiences. We achieve these results using only a few demonstration trajectories from non-expert users, who predominantly appreciate the intuitive demonstrat…
When Modalities Conflict: How Unimodal Reasoning Uncertainty Governs Preference Dynamics in MLLMs
Peking University（北京大学）；Provable Responsible AI and Data Analytics (PRADA) Lab（可证明负责任的人工智能和数据分析实验室）；King Abdullah University of Science and Technology（卡布斯大学）；South China University of Technology（华南理工大学）；Tsinghua University（清华大学）；University of Georgia（佐治亚大学）；MBZUAI（马克斯·普朗克人工智能研究所）
TUR-DPO: Topology- and Uncertainty-Aware Direct Preference Optimization
Artificial Intelligence and Innovation Centre, University of Kurdistan, Erbil, Iraq（人工智能与创新中心，乌尔米耶大学，伊拉克）；Department of Computer Engineering, University of Kurdistan, Iran（计算机工程系，乌尔米耶大学，伊朗）；Centre for Artificial Intelligence Research and Optimisation, Torrens University Australia, Brisbane, Australia（人工智能研究与优化中心，塔伦斯大学澳大利亚，布里斯班，澳大利亚）；Research and Innovation Center, Obuda University, Budapest 1034, Hungary（研究与创新中心，奥布达大学，布达佩斯1034，匈牙利）；Center for Machine Vision and Signal Analysis (CMVS), University of Oulu, Finland（机器视觉与信号分析中心（CMVS），奥卢大学，芬兰）
Reactive Multi-Robot Navigation in Outdoor Environments Through Uncertainty-Aware Active Learning of Human Preference Landscape
Kent State University（肯特州立大学）；Worcester Polytechnic Institute（伍斯特理工学院）；University of Pittsburgh（匹兹堡大学）；Northeastern University（东北大学）；College of Aeronautics and Engineering（航空航天与工程学院）
Why AI Safety Requires Uncertainty, Incomplete Preferences, and Non-Archimedean Utilities
School of Computer Science and Statistics, Trinity College Dublin（特里尼蒂大学计算机科学与统计学学院）；Trinity Quantum Alliance, Trinity College Dublin（特里尼蒂量子联盟）；Management in Networked and Digital Societies (MINDS) Department, Kozminski University（科津斯基大学网络化与数字化社会管理系）
Multi-agent reinforcement learning (MARL) has shown recent success in increasingly complex fixed-team zero-sum environments. However, the real world is not zero-sum nor does it have fixed teams; humans face numerous social dilemmas and must learn when to cooperate and when to compete. To successfully deploy agents into the human world, it may be important that they be able to understand and help in our conflicts. Unfortunately, selfish MARL agents typically fail when faced with social dilemmas. In this work, we show evidence of emergent direct reciprocity, indirect reciprocity and reputation, …
