Providing explanations is considered an imperative ability for an AI agent in a human-robot teaming framework. The right explanation provides the rationale behind an AI agent's decision-making. However, to maintain the human teammate's cognitive demand to comprehend the provided explanations, prior works have focused on providing explanations in a specific order or intertwining the explanation generation with plan execution. Moreover, these approaches do not consider the degree of details required to share throughout the provided explanations. In this work, we argue that the agent-generated explanations, especially the complex ones, should be abstracted to be aligned with the level of details the human teammate desires to maintain the recipient's cognitive load. Therefore, learning a hierarchical explanations model is a challenging task. Moreover, the agent needs to follow a consistent high-level policy to transfer the learned teammate preferences to a new scenario while lower-level detailed plans are different. Our evaluation confirmed the process of understanding an explanation, especially a complex and detailed explanation, is hierarchical. The human preference that reflected this aspect corresponded exactly to creating and employing abstraction for knowledge assimilation hidden deeper in our cognitive process. We showed that hierarchical explanations achieved better task performance and behavior interpretability while reduced cognitive load. These results shed light on designing explainable agents utilizing reinforcement learning and planning across various domains.
Prior work on generating explanations in a planning and decision-making context has focused on providing the rationale behind an AI agent's decision making. While these methods provide the right explanations from the explainer's perspective, they fail to heed the cognitive requirement of understanding an explanation from the explainee's (the human's) perspective. In this work, we set out to address this issue by first considering the influence of information order in an explanation, or the progressiveness of explanations. Intuitively, progression builds later concepts on previous ones and is k…
School of Computing and Augmented Intelligence, Arizona State University（计算与增强智能学院，亚利桑那州立大学）；School of Mathematical and Statistical Sciences, Arizona State University（数学与统计科学学院，亚利桑那州立大学）
Omni-Decision: Evidence-Ledger Planning for Omni-Modal Agents
Institute of Neuroscience, Chinese Academy of Sciences（中国科学院神经科学研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Tongyi Lab, Alibaba Group（阿里巴巴集团通义实验室）；Tsinghua University（清华大学）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）
Intelligent robots are redefining a multitude of critical domains but are still far from being fully capable of assisting human peers in day-to-day tasks. An important requirement of collaboration is for each teammate to maintain and respect an understanding of the others' expectations of itself. Lack of which may lead to serious issues such as loose coordination between teammates, reduced situation awareness, and ultimately teaming failures. Hence, it is important for robots to behave explicably by meeting the human's expectations. One of the challenges here is that the expectations of the hu…
Computer Science and Engineering Department at Arizona State University（亚利桑那州立大学计算机科学与工程系）
Generating explanation to explain its behavior is an essential capability for a robotic teammate. Explanations help human partners better understand the situation and maintain trust of their teammates. Prior work on robot generating explanations focuses on providing the reasoning behind its decision making. These approaches, however, fail to heed the cognitive requirement of understanding an explanation. In other words, while they provide the right explanations from the explainer's perspective, the explainee part of the equation is ignored. In this work, we address an important aspect along th…
Human-robot interaction can be divided into two categories based on the physical distance between the human and robot: remote and proximal. In proximal interaction, the human and robot often engage in close coordination; in remote interaction, the human and robot are less coupled due to communication constraints. As a result, providing automation for the robot in remote interaction becomes more important. Thus far, human factor studies on automation in remote human-robot interaction have been restricted to various forms of supervision, in which the robot is essentially being used as a smart mo…
GAP: Graph-Based Agent Planning with Parallel Tool Use and Reinforcement Learning
Tsinghua University（清华大学）；Huazhong University of Science and Technology（华中科技大学）；National University of Singapore（新加坡国立大学）；Carnegie Mellon University（卡内基梅隆大学）
Often, when modeling human decision-making behaviors in the context of human-robot teaming, the emotion aspect of human is ignored. Nevertheless, the influence of emotion, in some cases, is not only undeniable but beneficial. This work studies the human-like characteristics brought by regret emotion in one-human-multi-robot teaming for the application of domain search. In such application, the task management load is outsourced to the robots to reduce the human's workload, freeing the human to do more important work. The regret decision model is first used by each robot for deciding whether to…
X-Teaming: Multi-Turn Jailbreaks and Defenses with Adaptive Multi-Agents
University of California, Los Angeles（加州大学洛杉矶分校）；University of Washington（华盛顿大学）；Qatar Computing Research Institute（卡塔尔计算研究所）；Google（谷歌）；Stanford University（斯坦福大学）
An LLM-based Framework for Human-Swarm Teaming Cognition in Disaster Search and Rescue
School of Electronics and Information, Northwestern Polytechnical University（电子工程学院，西北工业大学）；Chongqing Institute for Brain and Intelligence, Guangyang Bay Laboratory（脑科学与智能研究院，广阳湾实验室）
Research on multi-agent planning has been popular in recent years. While previous research has been motivated by the understanding that, through cooperation, multi-agent systems can achieve tasks that are unachievable by single-agent systems, there are no formal characterizations of situations where cooperation is required to achieve a goal, thus warranting the application of multi-agent systems. In this paper, we provide such a formal discussion from the planning aspect. We first show that determining whether there is required cooperation (RC) is intractable is general. Then, by dividing the …
MINT: Minimal Information Neuro-Symbolic Tree for Objective-Driven Knowledge-Gap Reasoning and Active Elicitation
Department of Electrical and Computer Engineering, George Washington University（乔治华盛顿大学电气与计算机工程系）；Department of Electrical and Computer Engineering, Northeastern University（东北大学电气与计算机工程系）
IBM T. J. Watson Research Center（IBM T. J. 沃森研究中心）
In this paper, we introduce the problem of denoting and deriving the complexity of workflows (plans, schedules) in collaborative, planner-assisted settings where humans and agents are trying to jointly solve a task. The interactions -- and hence the workflows that connect the human and the agents -- may differ according to the domain and the kind of agents. We adapt insights from prior work in human-agent teaming and workflow analysis to suggest metrics for workflow complexity. The main motivation behind this work is to highlight metrics for human comprehensibility of plans and schedules. The …
PAGER: Bridging the Semantic-Execution Gap in Point-Precise Geometric GUI Control
University of Chinese Academy of Sciences（中国科学院大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；China University of Petroleum-Beijing（中国石油大学（北京））
Polytechnique Montreal（蒙特利尔理工学院）；NASA Jet Propulsion Laboratory, California Institute of Technology（美国国家航空航天局喷气推进实验室，加州理工学院）；University of Southern California（南加州大学）；University of California, Riverside（加州大学河滨分校）
Real-world deployment of new technology and capabilities can be daunting. The recent DARPA Subterranean (SubT) Challenge, for instance, aimed at the advancement of robotic platforms and autonomy capabilities in three one-year development pushes. While multi-agent systems are traditionally deployed in controlled and structured environments that allow for controlled testing (e.g., warehouses), the SubT challenge targeted various types of unknown underground environments that imposed the risk of robot loss in the case of failure. In this work, we introduce a video game-inspired interface, an auto…
Video prediction models combined with planning algorithms have shown promise in enabling robots to learn to perform many vision-based tasks through only self-supervision, reaching novel goals in cluttered scenes with unseen objects. However, due to the compounding uncertainty in long horizon video prediction and poor scalability of sampling-based planning optimizers, one significant limitation of these approaches is the ability to plan over long horizons to reach distant goals. To that end, we propose a framework for subgoal generation and planning, hierarchical visual foresight (HVF), which g…
Unifying Object-Centric World Models and Diffusion Policy: A Hierarchical Framework for Multi-Stage Robotic Tasks
Tandon School of Engineering, New York University（纽约大学坦登工程学院）；Courant Institute of Mathematical Sciences, New York University（纽约大学库朗数学科学研究所）；AMI Labs（AMI实验室）
DIRECT: Video Mashup Creation via Hierarchical Multi-Agent Planning and Intent-Guided Editing
School of Electronics Engineering and Computer Science, Peking University（北京大学电子工程与计算机科学学院）；School of Computer Science, Peking University（北京大学计算机科学学院）；Huazhong University of Science and Technology（华中科技大学）
UniConFlow: A Unified Constrained Flow-Matching Framework for Certified Motion Planning
Technical University of Munich (TUM)（技术大学慕尼黑）；School of Mechanical Engineering, Translational Research Center, Tongji University（机械工程学院、转化研究中心，同济大学）；Mohamed Bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）
HygieneRoboBench: Benchmarking Hygiene-Aware Planning for Household Robots
Shanghai Jiao Tong University（上海交通大学）；Eastern Institute of Technology, Ningbo（宁波东方理工大学）；University of Waterloo（滑铁卢大学）；Stony Brook University（纽约州立大学石溪分校）；University of Science and Technology of China（中国科学技术大学）；Chengdu Institute of Computer Applications, Chinese Academy of Sciences（中国科学院成都计算机应用研究所）；Ningbo Institute of Digital Twin（宁波数字孪生研究院）
