ILeSiA: Interactive Learning of Robot Situational Awareness from Camera Input
Czech Institute of Informatics, Robotics, and Cybernetics, Czech Technical University in Prague, Czech Republic（捷克信息学、机器人学与自动控制研究所，布拉格捷克技术大学）；Cognitive Robotics, Delft University of Technology, The Netherlands（认知机器人学，代尔夫特理工大学）
Petr Vanc, Giovanni Franzese, Jan Kristof Behrens, Cosimo Della Santina, Karla Stepanova, Jens Kober, Robert Babuska
备注：
8 pages, 9 figures. IEEE Robotics and Automation Letters. Accepted August 2025
Learning from demonstration is a promising approach for teaching robots new skills. However, a central challenge in the execution of acquired skills is the ability to recognize faults and prevent failures. This is essential because demonstrations typically cover only a limited set of scenarios and often only the successful ones. During task execution, unforeseen situations may arise, such as changes in the robot's environment or interaction with human operators. To recognize such situations, this paper focuses on teaching the robot situational awareness by using a camera input and labeling frames as safe or risky. We train a Gaussian Process (GP) regression model fed by a low-dimensional latent space representation of the input images. The model outputs a continuous risk score ranging from zero to one, quantifying the degree of risk at each timestep. This allows for pausing task execution in unsafe situations and directly adding new training data, labeled by the human user. Our experiments on a robotic manipulator show that the proposed method can reliably detect both known and novel faults using only a single example for each new fault. In contrast, a standard multi-layer perceptron (MLP) performs well only on faults it has encountered during training. Our method enables the next generation of cobots to be rapidly deployed with easy-to-set-up, vision-based risk assessment, proactively safeguarding humans and detecting misaligned parts or missing objects before failures occur. We provide all the code and data required to reproduce our experiments at imitrob.ciirc.cvut.cz/publications/ilesia.
This paper describes a technique for the autonomous mission planning of unmanned aerial system swarms. Given a swarm operating in a known area, a central command system generates measurements from the swarm. If those measurements indicate changes to the mission situation such as target movement, the swarm planning is updated to reflect the new situation and guidance updates are broadcast to the swarm. The primary algorithms featured in this work are A* pathfinding and the Generalized Labeled Multi-Bernoulli multi-target tracking method.
Alternative Interfaces for Human-initiated Natural Language Communication and Robot-initiated Haptic Feedback: Towards Better Situational Awareness in Human-Robot Collaboration
SymAware: A Software Development Framework for Trustworthy Multi-Agent Systems with Situational Awareness
School of Computing, Newcastle University（纽卡斯尔大学计算机学院）；Eindhoven University of Technology（埃因霍温理工大学）；KTH Royal Institute of Technology（瑞典皇家理工学院）；Max Planck Institute for software systems（马克斯·普朗克软件系统研究所）
Scania CV AB（斯堪尼亚CV AB公司）；KTH Royal Institute of Technology（瑞典皇家理工学院）
One of the main challenges in developing autonomous transport systems based on connected and automated vehicles is the comprehension and understanding of the environment around each vehicle. In many situations, the understanding is limited to the information gathered by the sensors mounted on the ego-vehicle, and it might be severely affected by occlusion caused by other vehicles or fixed obstacles along the road. Situational awareness is the ability to perceive and comprehend a traffic situation and to predict the intent of vehicles and road users in the surrounding of the ego-vehicle. The ma…
This paper studies how global dynamics and knowledge of high-level features can inform decision-making for robots in flow-like environments. Specifically, we investigate how coherent sets, an environmental feature found in these environments, inform robot awareness within these scenarios. The proposed approach is an online environmental feature generator which can be used for robot reasoning. We compute coherent sets online with techniques from machine learning and design frameworks for robot behavior that leverage coherent set features. We demonstrate the effectiveness of online methods over …
This paper describes a technique for the autonomous mission planning of robotic swarms in high risk environments where agent disablement is likely. Given a swarm operating in a known area, a central command system generates measurements from the swarm. If those measurements indicate changes to the mission situation such as target movement or agent loss, the swarm planning is updated to reflect the new situation and guidance updates are broadcast to the swarm. The primary algorithms featured in this work are A* pathfinding and the Generalized Labeled Multi-Bernoulli multi-object tracking method…
Tracking and locating radio-tagged wildlife is a labor-intensive and time-consuming task necessary in wildlife conservation. In this article, we focus on the problem of achieving embedded autonomy for a resource-limited aerial robot for the task capable of avoiding undesirable disturbances to wildlife. We employ a lightweight sensor system capable of simultaneous (noisy) measurements of radio signal strength information from multiple tags for estimating object locations. We formulate a new lightweight task-based trajectory planning method-LAVAPilot-with a greedy evaluation strategy and a void …
An Exploratory Study on Human-Robot Interaction using Semantics-based Situational Awareness
Extreme Robotics Lab (ERL) and National Center for Nuclear Robotics (NCNR), University of Birmingham, UK（极端机器人实验室（ERL）和核机器人国家中心（NCNR）、英国伯明翰大学）；School of Electronic Engineering and Computer Science, Queen Mary University of London, UK（电子工程与计算机科学学院，伦敦女王玛丽大学）
The majority of fatalities and traumatic injuries in heavy industries involve mobile plant and vehicles, often resulting from a lapse of attention or communication. Existing approaches to hazard identification include the use of human spotters, passive reversing cameras, non-differentiating proximity sensors and tag based systems. These approaches either suffer from problems of worker attention or require the use of additional devices on all workers and obstacles. Whilst computer vision detection systems have previously been deployed in structured applications such as manufacturing and on-road…
First Responders' Perceptions of Semantic Information for Situational Awareness in Robot-Assisted Emergency Response
Extreme Robotics Lab (ERL) and National Center for Nuclear Robotics (NCNR), University of Birmingham, UK（极端机器人实验室（ERL）和核机器人国家中心（NCNR）、伯明翰大学）；Department of Computer Science, Bioengineering, Robotics and Systems Engineering, University of Genova, Italy（计算机科学、生物工程、机器人学和系统工程系、热那亚大学）；School of Engineering Mathematics and Technology, University of Bristol, UK（工程数学与技术学院、布里斯托尔大学）；School of Electronic Engineering and Computer Science, Queen Mary University of London, UK（电子工程与计算机科学学院、伦敦女王大学）
A Framework for Semantics-based Situational Awareness during Mobile Robot Deployments
University of Birmingham（伯明翰大学）；Birmingham Centre for Nuclear Education & Research (BCNER)（伯明翰核教育研究中心）；University of West Attica（西阿提卡大学）；Queen Mary University of London（伦敦大学玛丽皇后学院）
Universidad Politécnica de Madrid（马德里理工大学）；Centre for Automation and Robotics (CAR), Universidad Politécnica de Madrid (UPM-CSIC)（马德里理工大学自动化与机器人中心（UPM-CSIC））；University of Luxembourg（卢森堡大学）
Democritus University of Thrace（色雷斯德谟克利特大学）；Information Technologies Institute, The Centre for Research & Technology, Hellas（信息技术研究所，研究与技术中心，希腊）
This paper tackles the problem of positioning a swarm of UAVs inside a completely unknown terrain, having as objective to maximize the overall situational awareness. The situational awareness is expressed by the number and quality of unique objects of interest, inside the UAVs' fields of view. YOLOv3 and a system to identify duplicate objects of interest were employed to assign a single score to each UAVs' configuration. Then, a novel navigation algorithm, capable of optimizing the previously defined score, without taking into consideration the dynamics of either UAVs or environment, is propos…
This paper addresses task-allocation problems with uncertainty in situational awareness for distributed autonomous robots (DARs). The uncertainty propagation over a task-allocation process is done by using the Unscented transform that uses the Sigma-Point sampling mechanism. It has great potential to be employed for generic task-allocation schemes, in the sense that there is no need to modify an existing task-allocation method that has been developed without considering the uncertainty in the situational awareness. The proposed framework was tested in a simulated environment where the decision…
University of Luxembourg（卢森堡大学）；Universidad de Zaragoza（萨拉戈萨大学）
Mobile robots extract information from its environment to understand their current situation to enable intelligent decision making and autonomous task execution. In our previous work, we introduced the concept of Situation Graphs (S-Graphs) which combines in a single optimizable graph, the robot keyframes and the representation of the environment with geometric, semantic and topological abstractions. Although S-Graphs were built and optimized in real-time and demonstrated state-of-the-art results, they are limited to specific structured environments with specific hand-tuned dimensions of rooms…
