事件相机（或称神经形态相机）通过异步报告亮度的显著变化（即事件），提供了一种对自然场景的新颖编码方式；与传统相机相比，它具有更高的动态范围、时间分辨率以及更低的数据带宽。然而，其在特定领域研究任务中的采用受到阻碍，部分原因在于商业供应有限、现有数据集缺乏，以及预测其非线性光学编码、独特噪声模型和基于张量的数据处理需求所带来的挑战。为应对这些挑战，我们推出 Python 中的 Synthetic Events for Neural Processing and Integration（SENPI），这是一个基于 PyTorch 的库，用于模拟和处理事件相机数据。SENPI 包含一个可微数字孪生，可将基于强度的数据转换为事件表示，从而能够在处理前向模型非平滑、非线性特性的同时评估事件相机性能。该库还支持用于基于事件的 I/O、操作、滤波和可视化的模块，为合成和真实的基于事件的数据创建高效且可扩展的工作流。我们通过将合成输出与真实事件相机数据进行比较，证明了 SENPI 生成逼真的基于事件的数据的能力，并利用这些结果对基于事件的感知的特性和效用得出结论。此外，我们展示了 SENPI 在探索不同噪声条件下事件相机行为，以及优化事件对比度阈值以在目标条件下改善编码方面的用途。最终，SENPI 旨在通过提供一个易于使用的事件数据生成和算法开发工具，降低研究人员的入门门槛，使其成为推进神经形态视觉系统研究的宝贵资源。
英文摘要：
Event, or neuromorphic cameras, offer a novel encoding of natural scenes by asynchronously reporting significant changes in brightness, known as events, with improved dynamic range, temporal resolution and lower data bandwidth when compared to conventional cameras. However, their adoption in domain-specific research tasks is hindered in part by limited commercial availability, lack of existing datasets, and challenges related to predicting the impact of their nonlinear optical encoding, unique noise model and tensor-based data processing requirements. To address these challenges, we introduce Synthetic Events for Neural Processing and Integration (SENPI) in Python, a PyTorch-based library for simulating and processing event camera data. SENPI includes a differentiable digital twin that converts intensity-based data into event representations, allowing for evaluation of event camera performance while handling the non-smooth and nonlinear nature of the forward model The library also supports modules for event-based I/O, manipulation, filtering and visualization, creating efficient and scalable workflows for both synthetic and real event-based data. We demonstrate SENPI's ability to produce realistic event-based data by comparing synthetic outputs to real event camera data and use these results to draw conclusions on the properties and utility of event-based perception. Additionally, we showcase SENPI's use in exploring event camera behavior under varying noise conditions and optimizing event contrast threshold for improved encoding under target conditions. Ultimately, SENPI aims to lower the barrier to entry for researchers by providing an accessible tool for event data generation and algorithmic developmnent, making it a valuable resource for advancing research in neuromorphic vision systems.
Person re-identification (re-ID), which aims to re-identify people across different camera views, has been significantly advanced by deep learning in recent years, particularly with convolutional neural networks (CNNs). In this paper, we present Torchreid, a software library built on PyTorch that allows fast development and end-to-end training and evaluation of deep re-ID models. As a general-purpose framework for person re-ID research, Torchreid provides (1) unified data loaders that support 15 commonly used re-ID benchmark datasets covering both image and video domains, (2) streamlined pipel…
Noise2Image: Noise-Enabled Static Scene Recovery for Event Cameras
Department of Bioengineering, University of California, Berkeley（生物工程系，加州大学伯克利分校）；Department of Electrical Engineering and Computer Sciences, University of California, Berkeley（电气工程与计算机科学系，加州大学伯克利分校）；Department of Vision Science, University of California, Berkeley（视觉科学系，加州大学伯克利分校）
3D shape reconstruction is a primary component of augmented/virtual reality. Despite being highly advanced, existing solutions based on RGB, RGB-D and Lidar sensors are power and data intensive, which introduces challenges for deployment in edge devices. We approach 3D reconstruction with an event camera, a sensor with significantly lower power, latency and data expense while enabling high dynamic range. While previous event-based 3D reconstruction methods are primarily based on stereo vision, we cast the problem as multi-view shape from silhouette using a monocular event camera. The output fr…
Renault（雷诺）；LEAT / CNRS UMR 7248（LEAT / 法国国家科学研究中心 UMR 7248）；University Côte d’Azur（蔚蓝海岸大学）
Convolutional neural networks (CNNs) are now the de facto solution for computer vision problems thanks to their impressive results and ease of learning. These networks are composed of layers of connected units called artificial neurons, loosely modeling the neurons in a biological brain. However, their implementation on conventional hardware (CPU/GPU) results in high power consumption, making their integration on embedded systems difficult. In a car for example, embedded algorithms have very high constraints in term of energy, latency and accuracy. To design more efficient computer vision algo…
University of Missouri-Columbia（密苏里大学哥伦比亚分校）；WSP USA（美国WSP公司）
Manual traffic surveillance can be a daunting task as Traffic Management Centers operate a myriad of cameras installed over a network. Injecting some level of automation could help lighten the workload of human operators performing manual surveillance and facilitate making proactive decisions which would reduce the impact of incidents and recurring congestion on roadways. This article presents a novel approach to automatically monitor real time traffic footage using deep convolutional neural networks and a stand-alone graphical user interface. The authors describe the results of research recei…
Real-time visual analysis tasks, like tracking and recognition, require swift execution of computationally intensive algorithms. Visual sensor networks can be enabled to perform such tasks by augmenting the sensor network with processing nodes and distributing the computational burden in a way that the cameras contend for the processing nodes while trying to minimize their task completion times. In this paper, we formulate the problem of minimizing the completion time of all camera sensors as an optimization problem. We propose algorithms for fully distributed optimization, analyze the existen…
Traffic cameras are commonly deployed monitoring components in road infrastructure networks, providing operators visual information about conditions at critical points in the network. However, human observers are often limited in their ability to process simultaneous information sources. Recent advancements in computer vision, driven by deep learning methods, have enabled general object recognition, unlocking opportunities for camera-based sensing beyond the existing human observer paradigm. In this paper, we present a Natural Language Processing (NLP)-inspired approach, entitled Bag-of-Label-…
3D hand tracking methods based on monocular RGB videos are easily affected by motion blur, while event camera, a sensor with high temporal resolution and dynamic range, is naturally suitable for this task with sparse output and low power consumption. However, obtaining 3D annotations of fast-moving hands is difficult for constructing event-based hand-tracking datasets. In this paper, we provided an event-based speed adaptive hand tracker (ESAHT) to solve the hand tracking problem based on event camera. We enabled a CNN model trained on a hand tracking dataset with slow motion, which enabled th…
Chalmers University of Technology（查尔姆斯理工大学）；AB Volvo（沃尔沃集团）
Recent advances in the field of machine learning and computer vision have enabled the development of fast and accurate road detectors. Commonly such systems are trained within a supervised learning paradigm where both an input sensor's data and the corresponding ground truth label must be provided. The task of generating labels is commonly carried out by human annotators and it is notoriously time consuming and expensive. In this work, it is shown that a semi-supervised approach known as co-training can provide significant F1-score average improvements compared to supervised learning. In co-tr…
Solving the camera-to-robot pose is a fundamental requirement for vision-based robot control, and is a process that takes considerable effort and cares to make accurate. Traditional approaches require modification of the robot via markers, and subsequent deep learning approaches enabled markerless feature extraction. Mainstream deep learning methods only use synthetic data and rely on Domain Randomization to fill the sim-to-real gap, because acquiring the 3D annotation is labor-intensive. In this work, we go beyond the limitation of 3D annotations for real-world data. We propose an end-to-end …
Surround-view cameras are a primary sensor for automated driving, used for near-field perception. It is one of the most commonly used sensors in commercial vehicles primarily used for parking visualization and automated parking. Four fisheye cameras with a 190° field of view cover the 360° around the vehicle. Due to its high radial distortion, the standard algorithms do not extend easily. Previously, we released the first public fisheye surround-view dataset named WoodScape. In this work, we release a synthetic version of the surround-view dataset, covering many of its weaknesses and extending…
Deep learning techniques have enabled rapid progress in monocular depth estimation, but their quality is limited by the ill-posed nature of the problem and the scarcity of high quality datasets. We estimate depth from a single camera by leveraging the dual-pixel auto-focus hardware that is increasingly common on modern camera sensors. Classic stereo algorithms and prior learning-based depth estimation techniques under-perform when applied on this dual-pixel data, the former due to too-strong assumptions about RGB image matching, and the latter due to not leveraging the understanding of optics …
An AI-Enabled Framework Within Reach for Enhancing Healthcare Sustainability and Fairness
AI Research Center, Hangzhou Innovation Institute, Beihang University（北京航空航天大学杭州创新研究院AI研究中心）；School of Computer Science, Hangzhou Dianzi University（杭州电子科技大学计算机学院）；School of Automation Science and Electrical Engineering, Beihang University（北京航空航天大学自动化科学与电气工程学院）；Institute of Medical Technology, Health Science Center of Peking University（北京大学健康科学中心医学技术研究所）；National Institute of Health Data Science, Peking University（北京大学健康数据科学国家研究院）；Institute of Artificial Intelligence, Beihang University（北京航空航天大学人工智能研究院）；Information Hub, Hong Kong University of Science and Technology（香港科技大学信息中心）；Department of Biomedical Engineering, City University of Hong Kong（香港城市大学生物医学工程系）；Department of Biomedical Engineering, Southern University of Science and Technology（南方科技大学生物医学工程系）；Institute of Medical Information, Chinese Academy of Medical Sciences & Peking Union Medical College（中国医学科学院北京协和医学院医学信息研究所）
SemOD: Semantic Enabled Object Detection Network under Various Weather Conditions
Department of Electrical and Computer Engineering, University of Rochester（罗切斯特大学电气与计算机工程系）；Department of Electrical and Computer Engineering, University of California, Los Angeles（加州大学洛杉矶分校电气与计算机工程系）
