Recent trends have shown that autonomous agents, such as Autonomous Ground Vehicles (AGVs), Unmanned Aerial Vehicles (UAVs), and mobile robots, effectively improve human productivity in solving diverse tasks. However, since these agents are typically powered by portable batteries, they require extremely low power/energy consumption to operate in a long lifespan. To solve this challenge, neuromorphic computing has emerged as a promising solution, where bio-inspired Spiking Neural Networks (SNNs) use spikes from event-based cameras or data conversion pre-processing to perform sparse computations efficiently. However, the studies of SNN deployments for autonomous agents are still at an early stage. Hence, the optimization stages for enabling efficient embodied SNN deployments for autonomous agents have not been defined systematically. Toward this, we propose a novel framework called SNN4Agents that consists of a set of optimization techniques for designing energy-efficient embodied SNNs targeting autonomous agent applications. Our SNN4Agents employs weight quantization, timestep reduction, and attention window reduction to jointly improve the energy efficiency, reduce the memory footprint, optimize the processing latency, while maintaining high accuracy. In the evaluation, we investigate use cases of event-based car recognition, and explore the trade-offs among accuracy, latency, memory, and energy consumption. The experimental results show that our proposed framework can maintain high accuracy (i.e., 84.12% accuracy) with 68.75% memory saving, 3.58x speed-up, and 4.03x energy efficiency improvement as compared to the state-of-the-art work for NCARS dataset. In this manner, our SNN4Agents framework paves the way toward enabling energy-efficient embodied SNN deployments for autonomous agents.
University of Trieste（的里雅斯特大学）；Oslo Metropolitan University（奥斯陆城市大学）
Voxel-based Soft Robots (VSRs) are a form of modular soft robots, composed of several deformable cubes, i.e., voxels. Each VSR is thus an ensemble of simple agents, namely the voxels, which must cooperate to give rise to the overall VSR behavior. Within this paradigm, collective intelligence plays a key role in enabling the emerge of coordination, as each voxel is independently controlled, exploiting only the local sensory information together with some knowledge passed from its direct neighbors (distributed or collective control). In this work, we propose a novel form of collective control, i…
SpikingNav: Robust Embodied Navigation with Spiking Neural Policies
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；State Key Laboratory of Media Convergence and Communication, Communication University of China（中国传媒大学媒体融合与传播国家重点实验室）；Department of Computer Science and Technology, Tsinghua University（清华大学计算机科学与技术系）；Beijing National Research Center for Information Science and Technology, Tsinghua University（北京信息科学与技术国家研究中心）
Technische Universität Wien (TU Wien)（维也纳工业大学）；New York University Abu Dhabi (NYUAD)（纽约大学阿布扎比分校）
Recent advances have shown that SNN-based systems can efficiently perform unsupervised continual learning due to their bio-plausible learning rule, e.g., Spike-Timing-Dependent Plasticity (STDP). Such learning capabilities are especially beneficial for use cases like autonomous agents (e.g., robots and UAVs) that need to continuously adapt to dynamically changing scenarios/environments, where new data gathered directly from the environment may have novel features that should be learned online. Current state-of-the-art works employ high-precision weights (i.e., 32 bit) for both training and inf…
VL2Spike: Spike-driven Distillation from VLMs for Low-Power Visual Perception in Embodied AI
School of EEE, Nanyang Technological University (NTU)（南洋理工大学电气与电子工程学院）；Department of Computer Science, University of Toronto（多伦多大学计算机科学系）；Advanced Micro Devices, Inc.（超威半导体公司）；State Key Laboratory of Extreme Photonics and Instrumentation, Zhejiang University（浙江大学极端光子学与仪器国家重点实验室）；Faculty of Information Science and Technology, Hokkaido University（北海道大学信息科学与技术学院）
Université de Montéal（蒙特利尔大学）；Mila（米拉人工智能研究所）；ETH Zürich（苏黎世联邦理工学院）
In many situations it is either impossible or impractical to develop and evaluate agents entirely on the target domain on which they will be deployed. This is particularly true in robotics, where doing experiments on hardware is much more arduous than in simulation. This has become arguably more so in the case of learning-based agents. To this end, considerable recent effort has been devoted to developing increasingly realistic and higher fidelity simulators. However, we lack any principled way to evaluate how good a "proxy domain" is, specifically in terms of how useful it is in helping us ac…
Stanford University（斯坦福大学）；University of California, Berkeley（加州大学伯克利分校）
Developing visual perception models for active agents and sensorimotor control are cumbersome to be done in the physical world, as existing algorithms are too slow to efficiently learn in real-time and robots are fragile and costly. This has given rise to learning-in-simulation which consequently casts a question on whether the results transfer to real-world. In this paper, we are concerned with the problem of developing real-world perception for active agents, propose Gibson Virtual Environment for this purpose, and showcase sample perceptual tasks learned therein. Gibson is based on virtuali…
UnrealZoo: Enriching Photo-realistic Virtual Worlds for Embodied AI
School of Artificial Intelligence, Beijing Normal University（北京师范大学人工智能学院）；State Key Laboratory of Complex & Critical Software Environment, Beihang University（北航复杂与关键软件环境国家重点实验室）；School of Computer Science, Institute for Artificial Intelligence, State Key Laboratory of General Artificial Intelligence, Peking University（北京大学计算机学院）；City University of Macau（澳门城市大学）；National University of Singapore（新加坡国立大学）；Beijing Institute for General Artificial Intelligence (BIGAI)（北京通用人工智能研究院）
GenRL: Multimodal-foundation world models for generalization in embodied agents
IDLab, Ghent University（IDLab，根特大学）；VERSES AI Research Lab（VERSES AI研究实验室）；Mila, University of Montreal（米拉研究所，蒙特利尔大学）；ServiceNow Research（ServiceNow研究院）
Towards general embodied intelligence: integrating large language models, knowledge bases, and reasoning capabilities to build the next generation of AI agents
College of Mechanical Engineering, Chongqing University of Technology（重庆理工大学机械工程学院）；James Watt School of Engineering, University of Glasgow（格拉斯哥大学詹姆斯·瓦特工程学院）；School of Energy and Power, Jiangsu University of Science and Technology（江苏科技大学能源与动力学院）；Magnesium Research Center, Kumamoto University（熊本大学镁研究中心）；Department of Information and Communication Engineering, Nagoya University（名古屋大学信息与通信工程系）；State Key Laboratory of Fluid Power and Mechatronic Systems, Zhejiang University（浙江大学流体动力与机电系统国家重点实验室）
RoboView-Bias: Benchmarking Visual Bias in Embodied Agents for Robotic Manipulation
School of Cyber Science and Engineering, Nanjing University of Science and Technology（南京理工大学计算机科学与工程学院）；National University of Singapore（新加坡国立大学）；Sun Yat-sen University（中山大学）；Beihang University（北京航空航天大学）
Allen Institute for AI（艾伦人工智能研究所）；University of Washington（华盛顿大学）
There has been a significant recent progress in the field of Embodied AI with researchers developing models and algorithms enabling embodied agents to navigate and interact within completely unseen environments. In this paper, we propose a new dataset and baseline models for the task of Rearrangement. We particularly focus on the task of Room Rearrangement: an agent begins by exploring a room and recording objects' initial configurations. We then remove the agent and change the poses and states (e.g., open/closed) of some objects in the room. The agent must restore the initial configurations o…
