Being able to carry out complicated vision language reasoning tasks in 3D space represents a significant milestone in developing household robots and human-centered embodied AI. In this work, we demonstrate that a critical and distinct challenge in 3D vision language reasoning is situational awareness, which incorporates two key components: (1) The autonomous agent grounds its self-location based on a language prompt. (2) The agent answers open-ended questions from the perspective of its calculated position. To address this challenge, we introduce SIG3D, an end-to-end Situation-Grounded model for 3D vision language reasoning. We tokenize the 3D scene into sparse voxel representation and propose a language-grounded situation estimator, followed by a situated question answering module. Experiments on the SQA3D and ScanQA datasets show that SIG3D outperforms state-of-the-art models in situation estimation and question answering by a large margin (e.g., an enhancement of over 30% on situation estimation accuracy). Subsequent analysis corroborates our architectural design choices, explores the distinct functions of visual and textual tokens, and highlights the importance of situational awareness in the domain of 3D question answering.
University of Arizona（亚利桑那大学）；University of Texas at Austin（德克萨斯大学奥斯汀分校）
Space situational awareness typically makes use of physical measurements from radar, telescopes, and other assets to monitor satellites and other spacecraft for operational, navigational, and defense purposes. In this work we explore using textual input for the space situational awareness task. We construct a corpus of 48.5k news articles spanning all known active satellites between 2009 and 2020. Using a dependency-rule-based extraction system designed to target three high-impact events -- spacecraft launches, failures, and decommissionings, we identify 1,787 space-event sentences that are th…
This paper addresses task-allocation problems with uncertainty in situational awareness for distributed autonomous robots (DARs). The uncertainty propagation over a task-allocation process is done by using the Unscented transform that uses the Sigma-Point sampling mechanism. It has great potential to be employed for generic task-allocation schemes, in the sense that there is no need to modify an existing task-allocation method that has been developed without considering the uncertainty in the situational awareness. The proposed framework was tested in a simulated environment where the decision…
University of Luxembourg（卢森堡大学）；Universidad de Zaragoza（萨拉戈萨大学）
Mobile robots extract information from its environment to understand their current situation to enable intelligent decision making and autonomous task execution. In our previous work, we introduced the concept of Situation Graphs (S-Graphs) which combines in a single optimizable graph, the robot keyframes and the representation of the environment with geometric, semantic and topological abstractions. Although S-Graphs were built and optimized in real-time and demonstrated state-of-the-art results, they are limited to specific structured environments with specific hand-tuned dimensions of rooms…
Beijing Institute of Aerospace Systems Engineering（北京宇航系统工程研究所）；Space Engineering University（航天工程大学）；China Academy of Launch Vehicle Technology（中国运载火箭技术研究院）
Event camera is a new type of sensor that is different from traditional cameras. Each pixel is triggered asynchronously by an event. The trigger event is the change of the brightness irradiated on the pixel. If the increment or decrement is higher than a certain threshold, the event is output. Compared with traditional cameras, event cameras have the advantages of high temporal resolution, low latency, high dynamic range, low bandwidth and low power consumption. We carried out a series of observation experiments in a simulated space lighting environment. The experimental results show that the …
Autonomous vehicles are expected to operate safely in real-life road conditions in the next years. Nevertheless, unanticipated events such as the existence of unexpected objects in the range of the road, can put safety at risk. The advancement of sensing and communication technologies and Internet of Things may facilitate the recognition of hazardous situations and information exchange in a cooperative driving scheme, providing new opportunities for the increase of collaborative situational awareness. Safe and unobtrusive visualization of the obtained information may nowadays be enabled throug…
School of Computer Science and Engineering, Nanyang Technological University（南洋理工大学计算机科学与工程学院）；Rolls-Royce Plc（劳斯莱斯公司）；Rolls-Royce Derby（劳斯莱斯德比）
This paper presents a multi-sensor architecture with an adaptive multi-sensor management system suitable for control and navigation of autonomous maritime vessels in hazy and poor-visibility conditions. This architecture resides in the autonomous maritime vessels. It augments the data from on-board imaging sensors and weather sensors with the AIS data and weather data from sensors on other vessels and the on-shore vessel traffic surveillance system. The combined data is analyzed using computational intelligence and data analytics to determine suitable course of action while utilizing historica…
NavEYE: Vision-Centered Multi-Sensor Fusion-Based Situational Awareness System for Intelligent Surface Vehicles
Sanya Science and Education Innovation Park, Wuhan University of Technology（武汉理工大学三亚科教创新园）；School of Navigation, Wuhan University of Technology（武汉理工大学航运学院）；State Key Laboratory of Maritime Technology and Safety（船舶技术与安全国家重点实验室）
We present a new four-pronged approach to build firefighter's situational awareness for the first time in the literature. We construct a series of deep learning frameworks built on top of one another to enhance the safety, efficiency, and successful completion of rescue missions conducted by firefighters in emergency first response settings. First, we used a deep Convolutional Neural Network (CNN) system to classify and identify objects of interest from thermal imagery in real-time. Next, we extended this CNN framework for object detection, tracking, segmentation with a Mask RCNN framework, an…
International Centre for Neuromorphic Systems（国际神经形态系统中心）；The MARCS Institute for Brain, Behaviour, and Development（MARCS大脑、行为与发展研究所）；Western Sydney University（西悉尼大学）
In this work, we present optical space imaging using an unconventional yet promising class of imaging devices known as neuromorphic event-based sensors. These devices, which are modeled on the human retina, do not operate with frames, but rather generate asynchronous streams of events in response to changes in log-illumination at each pixel. These devices are therefore extremely fast, do not have fixed exposure times, allow for imaging whilst the device is moving and enable low power space imaging during daytime as well as night without modification of the sensors. Recorded at multiple remote …
ILeSiA: Interactive Learning of Robot Situational Awareness from Camera Input
Czech Institute of Informatics, Robotics, and Cybernetics, Czech Technical University in Prague, Czech Republic（捷克信息学、机器人学与自动控制研究所，布拉格捷克技术大学）；Cognitive Robotics, Delft University of Technology, The Netherlands（认知机器人学，代尔夫特理工大学）
Situational awareness and Indoor location tracking for firefighters is one of the tasks with paramount importance in search and rescue operations. For Indoor Positioning systems (IPS), GPS is not the best possible solution. There are few other techniques like dead reckoning, Wifi and bluetooth based triangulation, Structure from Motion (SFM) based scene reconstruction for Indoor positioning system. However due to high temperatures, the rapidly changing environment of fires, and low parallax in the thermal images, these techniques are not suitable for relaying the necessary information in a fir…
Enhancing Maritime Situational Awareness through End-to-End Onboard Raw Data Analysis
European Space Agency（欧洲空间局）；University of Naples Federico II（那不勒斯费德里科二世大学）；Ben-Gurion University of the Negev（本-古里安大学内盖夫分校）；University of Campania “L. Vanvitelli”（坎帕尼亚大学“L.万维泰利”分校）；Advanced Concepts and Studies Office, European Space Agency（欧洲空间局先进概念与研究办公室）
Situational awareness as a necessity in the connected and autonomous vehicles (CAV) domain is the subject of a significant number of researches in recent years. The driver's safety is directly dependent on the robustness, reliability, and scalability of such systems. Cooperative mechanisms have provided a solution to improve situational awareness by utilizing high speed wireless vehicular networks. These mechanisms mitigate problems such as occlusion and sensor range limitation. However, the network capacity is a factor determining the maximum amount of information being shared among cooperati…
