Despite the success of neural networks in computer vision tasks, digital 'neurons' are a very loose approximation of biological neurons. Today's learning approaches are designed to function on digital devices with digital data representations such as image frames. In contrast, biological vision systems are generally much more capable and efficient than state-of-the-art digital computer vision algorithms. Event cameras are an emerging sensor technology which imitates biological vision with asynchronously firing pixels, eschewing the concept of the image frame. To leverage modern learning techniques, many event-based algorithms are forced to accumulate events back to image frames, somewhat squandering the advantages of event cameras.
We follow the opposite paradigm and develop a new type of neural network which operates closer to the original event data stream. We demonstrate state-of-the-art performance in angular velocity regression and competitive optical flow estimation, while avoiding difficulties related to training SNN. Furthermore, the processing latency of our proposed approach is less than 1/10 any other implementation, while continuous inference increases this improvement by another order of magnitude.
Towards Low-Latency Event Stream-based Visual Object Tracking: A Slow-Fast Approach
School of Computer Science and Technology, Anhui University(安徽大学计算机科学与技术学院);Beijing Institute of Technology(北京理工大学);School of Electronic and Information Engineering, Anhui University(安徽大学电子与信息工程学院);Peng Cheng Laboratory(鹏城实验室);School of Computer Science, Peking University(北京大学计算机科学学院);School of Electronic and Computer Engineering, Shenzhen Graduate School, Peking University(北京大学深圳研究生院电子与计算机工程学院)
Hardware-Accelerated Event-Graph Neural Networks for Low-Latency Time-Series Classification on SoC FPGA
Graduate School of Science and Technology(科学与技术研究生院);Embedded Vision Systems Group, Computer Vision Laboratory(嵌入式视觉系统组,计算机视觉实验室);AGH University of Krakow(克拉科夫AGH大学);CEA-List, Université Grenoble Alpes(CEA-List,格勒诺布尔阿尔卑斯大学)
Graph Neural Assisted Actor-Critic for Latency-Efficient Edge Vision System
CISTER Research Center(CISTER研究中心);Instituto de Telecomunicações(电信研究所);Faculdade de Engenharia, Universidade do Porto(波尔图大学工程学院);Kennesaw State University(肯尼索州立大学);OmniVision Technologies Inc.(豪威科技公司);Pontificia Universidad Católica de Chile(智利天主教大学)
VisualThink-VLA: Visual Intermediate Reasoning for Effective and Low-Latency Vision-Language-Action Policies
Zhejiang University(浙江大学);Cornell University(康奈尔大学);National University of Singapore(新加坡国立大学);Xi'an University of Electronic Science and Technology(西安电子科技大学)
ViTMAlis: Towards Latency-Critical Mobile Video Analytics with Vision Transformers
School of Computing Science, Simon Fraser University(计算科学系,西蒙·弗雷泽大学);Global College, Shanghai Jiao Tong University(全球学院,上海交通大学);School of Science and Engineering (SSE) and The Future Network of Intelligence Institute (FNii), The Chinese University of Hong Kong, Shenzhen(科学与工程学院(SSE)和未来智能网络研究所(FNii),香港中文大学(深圳));Computer Science Department, University of Illinois Urbana-Champaign(计算机科学系,伊利诺伊大学厄巴纳-香槟分校)
State Key Laboratory for Novel Software Technology, Nanjing University(南京大学计算机软件新技术国家重点实验室);Institute for AI Industry Research (AIR), Tsinghua University(清华大学人工智能产业研究院);Shanghai AI Laboratory(上海人工智能实验室);Beijing Academy of Artificial Intelligence (BAAI)(北京人工智能研究院)
Efficient Partitioning Vision Transformer on Edge Devices for Distributed Inference
School of Computing, National University of Singapore(国家大学学院计算机科学学院);Information and Artificial Intelligence Institute, Zhejiang University of Finance & Economics Dongfang College(浙江财经大学东方学院信息与人工智能研究院);Department of Computer Science, ETH Zurich(苏黎世联邦理工学院计算机科学系);College of Computer Science and Technology, Zhejiang University(浙江大学计算机科学与技术学院);Lumia Ltd.(Lumia有限公司);Institute of Data Science, National University of Singapore(国家大学数据科学研究所)
Technische Universität Berlin(柏林工业大学);ETH Zurich(苏黎世联邦理工学院);University of Zurich(苏黎世大学);Intel Labs(英特尔实验室);Italian Institute of Technology(意大利理工学院);IBM Research(IBM研究院);Imperial College London(伦敦帝国理工学院);KTH Royal Institute of Technology(瑞典皇家理工学院);University of Pennsylvania(宾夕法尼亚大学)
事件相机是生物启发的传感器,与传统的帧相机不同:它们不是以固定的速率捕获图像,而是异步测量每像素的亮度变化,并输出一系列事件,对亮度变化的时间、位置和符号进行编码。与传统摄像机相比,事件摄像机提供了吸引人的特性:高时间分辨率(微秒量级)、非常高的动态范围(140 dB vs. 60 dB)、低功耗和高像素带宽(kHz量级),从而减少了运动模糊。因此,事件摄像机在传统摄像机的挑战性场景中具有很大的机器人和计算机视觉潜力,例如低延迟,高速和高动态范围。然而,需要新的方法来处理这些传感器的非常规输出,以释放其潜力。本文对基于事件的视觉这一新兴领域进行了全面概述,重点介绍了为解锁事件摄像机的出色特性而开发的应用程序和算法。我们从它们的工作原理,可用的实际传感器和它们已用于的任务,从低层次视觉(特征检测和跟踪,光流等)到高级视觉(重建、分割、识别)。我们还讨论了处理事件的技术,包括基于学习的技术,以及这些新传感器的专用处理器,如脉冲神经网络。此外,我们强调了仍有待解决的挑战和未来的机遇,以寻找一种更有效的、生物启发的方式,让机器感知世界并与世界互动。
Bend the Clock: Predicting Ahead to Beat Latency in Event-Based Object Detection
National University of Singapore(新加坡国立大学);IPAL CNRS IRL 2955(法国国家科研中心IPAL联合研究实验室2955);CerCo, CNRS UMR 5549, Université de Toulouse(法国国家科研中心CerCo实验室UMR 5549,图卢兹大学);ETIS UMR8051, CY Cergy Paris Université, ENSEA, CNRS(法国国家科研中心ETIS实验室UMR8051,CY塞尔吉巴黎大学,ENSEA)