Rethinking High-speed Image Reconstruction Framework with Spike Camera
School of Computer Science, Peking University（北京大学计算机学院）；State Key Laboratory for Multimedia Information Processing, Peking University（北京大学多媒体信息处理国家重点实验室）；Institute for Artificial Intelligence, Peking University（北京大学人工智能研究院）
Spike cameras, as innovative neuromorphic devices, generate continuous spike streams to capture high-speed scenes with lower bandwidth and higher dynamic range than traditional RGB cameras. However, reconstructing high-quality images from the spike input under low-light conditions remains challenging. Conventional learning-based methods often rely on the synthetic dataset as the supervision for training. Still, these approaches falter when dealing with noisy spikes fired under the low-light environment, leading to further performance degradation in the real-world dataset. This phenomenon is primarily due to inadequate noise modelling and the domain gap between synthetic and real datasets, resulting in recovered images with unclear textures, excessive noise, and diminished brightness. To address these challenges, we introduce a novel spike-to-image reconstruction framework SpikeCLIP that goes beyond traditional training paradigms. Leveraging the CLIP model's powerful capability to align text and images, we incorporate the textual description of the captured scene and unpaired high-quality datasets as the supervision. Our experiments on real-world low-light datasets U-CALTECH and U-CIFAR demonstrate that SpikeCLIP significantly enhances texture details and the luminance balance of recovered images. Furthermore, the reconstructed images are well-aligned with the broader visual features needed for downstream tasks, ensuring more robust and versatile performance in challenging environments.
Spike-NeRF: Neural Radiance Field Based On Spike Camera
National Engineering Research Center of Visual Technology (NERCVT), Peking University（北京大学视觉技术国家工程研究中心）；College of Future Technology, Peking University（北京大学未来技术学院）
Although recent learning-based calibration methods can predict extrinsic and intrinsic camera parameters from a single image, the accuracy of these methods is degraded in fisheye images. This degradation is caused by mismatching between the actual projection and expected projection. To address this problem, we propose a generic camera model that has the potential to address various types of distortion. Our generic camera model is utilized for learning-based methods through a closed-form numerical calculation of the camera projection. Simultaneously to recover rotation and fisheye distortion, w…
Depth estimation is essential for various important real-world applications such as autonomous driving. However, it suffers from severe performance degradation in high-velocity scenario since traditional cameras can only capture blurred images. To deal with this problem, the spike camera is designed to capture the pixel-wise luminance intensity at high frame rate. However, depth estimation with spike camera remains very challenging using traditional monocular or stereo depth estimation algorithms, which are based on the photometric consistency. In this paper, we propose a novel Uncertainty-Gui…
Tsinghua University（清华大学）；Huawei Inc.（华为公司）；University of Science and Technology of China（中国科学技术大学）；Hefei University of Technology（合肥工业大学）
The fundamental difficulty in person re-identification (ReID) lies in learning the correspondence among individual cameras. It strongly demands costly inter-camera annotations, yet the trained models are not guaranteed to transfer well to previously unseen cameras. These problems significantly limit the application of ReID. This paper rethinks the working mechanism of conventional ReID approaches and puts forward a new solution. With an effective operator named Camera-based Batch Normalization (CBN), we force the image data of all cameras to fall onto the same subspace, so that the distributio…
University of Washington（华盛顿大学）；Zhejiang University（浙江大学）；Silkwave Holdings Limited（Silkwave控股有限公司）
Radar has long been a common sensor on autonomous vehicles for obstacle ranging and speed estimation. However, as a robust sensor to all-weather conditions, radar's capability has not been well-exploited, compared with camera or LiDAR. Instead of just serving as a supplementary sensor, radar's rich information hidden in the radio frequencies can potentially provide useful clues to achieve more complicated tasks, like object classification and detection. In this paper, we propose a new dataset, named CRUW, with a systematic annotator and performance evaluation system to address the radar object…
We propose a CNN-based approach for multi-camera markerless motion capture of the human body. Unlike existing methods that first perform pose estimation on individual cameras and generate 3D models as post-processing, our approach makes use of 3D reasoning throughout a multi-stage approach. This novelty allows us to use provisional 3D models of human pose to rethink where the joints should be located in the image and to recover from past mistakes. Our principled refinement of 3D human poses lets us make use of image cues, even from images where we previously misdetected joints, to refine our e…
Cameras were originally designed using physics-based heuristics to capture aesthetic images. In recent years, there has been a transformation in camera design from being purely physics-driven to increasingly data-driven and task-specific. In this paper, we present a framework to understand the building blocks of this nascent field of end-to-end design of camera hardware and algorithms. As part of this framework, we show how methods that exploit both physics and data have become prevalent in imaging and computer vision, underscoring a key trend that will continue to dominate the future of task-…
High-speed and High-quality Vision Reconstruction of Spike Camera with Spike Stability Theorem
Pengcheng Laboratory（鹏城实验室）；School of Computer Science and Engineering, Sun Yat-sen University（中山大学计算机科学与工程学院）；School of Computer Science, Peking University（北京大学计算机学院）
Peking University（北京大学）；Capital Medical University（首都医科大学）
Spike camera mimicking the retina fovea can report per-pixel luminance intensity accumulation by firing spikes. As a bio-inspired vision sensor with high temporal resolution, it has a huge potential for computer vision. However, the sampling model in current Spike camera is so susceptible to quantization and noise that it cannot capture the texture details of objects effectively. In this work, a robust visual sampling model inspired by receptive field (RVSM) is proposed where wavelet filter generated by difference of Gaussian (DoG) and Gaussian filter are used to simulate receptive field. Usin…
SpikeGS: Learning 3D Gaussian Fields from Continuous Spike Stream
College of Computer Science, Chongqing University, Chongqing, China（重庆大学计算机学院）；Motovis Co., Ltd., Shanghai, China（Motovis公司）；University of Chinese Academy of Sciences, Beijing, China（中国科学院大学）；Mobile Perception Lab, ShanghaiTech University, Shanghai, China（上海科技大学移动感知实验室）
Peking University（北京大学）；Beijing Academy of Artificial Intelligence（北京人工智能研究院）
As a bio-inspired sensor with high temporal resolution, the spiking camera has an enormous potential in real applications, especially for motion estimation in high-speed scenes. However, frame-based and event-based methods are not well suited to spike streams from the spiking camera due to the different data modalities. To this end, we present, SCFlow, a tailored deep learning pipeline to estimate optical flow in high-speed scenes from spike streams. Importantly, a novel input representation is introduced which can adaptively remove the motion blur in spike streams according to the prior motio…
Conventional image signal processing (ISP) frameworks are designed to reconstruct an RGB image from a single raw measurement. As multi-camera systems become increasingly popular these days, it is worth exploring improvements in ISP frameworks by incorporating raw measurements from multiple cameras. This manuscript is an intermediate progress report of a new ISP framework that is under development, StereoISP. It employs raw measurements from a stereo camera pair to generate a demosaicked, denoised RGB image by utilizing disparity estimated between the two views. We investigate StereoISP by test…
Rethinking Efficient and Effective Point-based Networks for Event Camera Classification and Regression: EventMamba
MICS Thrust at the Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州）MICS分支）；Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ)（广东省人工智能与数字经济实验室（深圳））；School of Microelectronics at Southern University of Science and Technology（南方科技大学微电子学院）
Radar Guided Camera Verification for Automatic Emergency Braking Rethinking Object Detection in Radar Camera Fusion
Autonomous Systems Lab, Department of Mechatronics Engineering, SRM Institute of Science and Technology（SRM科学与技术学院机电工程系自主系统实验室）；Centre for Electric Mobility, Department of EEE, SRM Institute of Science and Technology（SRM科学与技术学院电气与电子工程系电动出行中心）
SOTA: Spike-Navigated Optimal TrAnsport Saliency Region Detection in Composite-bias Videos
State Key Laboratory for Multimedia Information Processing（多媒体信息处理国家重点实验室）；Hubei Key Laboratory of Transportation Internet of Things（湖北省交通运输物联网重点实验室）；Institute for Artificial Intelligence（人工智能研究院）
GeoMotion: Rethinking Motion Segmentation via Latent 4D Geometry
College of Computer Science and Technology, Zhejiang University of Technology（浙江工业大学计算机科学与技术学院）；Zhejiang Key Laboratory of Visual Information Intelligent Processing（浙江省视觉信息智能处理重点实验室）；State Key Lab of CAD & CG, Zhejiang University（CAD与CG国家重点实验室）
SpikeStereoNet: A Brain-Inspired Framework for Stereo Depth Estimation from Spike Streams
National Key Laboratory for Multimedia Information Processing, School of Computer Science, Peking University（北京大学计算机学院多媒体信息处理全国重点实验室）；Center on Frontiers of Computing Studies, School of Computer Science, Peking University（北京大学计算机学院计算前沿研究中心）；Institute for Artificial Intelligence, Peking University（北京大学人工智能研究院）；School of Engineering and Applied Science, University of Pennsylvania（宾夕法尼亚大学工程与应用科学学院）
A Joint Visual Compression and Perception Framework for Neuralmorphic Spiking Camera
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；National Engineering Research Center of Visual Technology, Peking University（北京大学视觉技术国家工程研究中心）；Wangxuan Institute of Computer Technology, Peking University（北京大学王选计算机技术研究所）
USP-Gaussian: Unifying Spike-based Image Reconstruction, Pose Correction and Gaussian Splatting
School of Computer Science, Peking University（北京大学计算机科学学院）；State Key Laboratory for Multimedia Information Processing, Peking University（北京大学多媒体信息处理国家重点实验室）；Institute for Artificial Intelligence, Peking University（北京大学人工智能研究院）
The Pennsylvania State University（宾夕法尼亚州立大学）；The Johns Hopkins University（约翰斯·霍普金斯大学）
Promising results have been achieved in image classification problems by exploiting the discriminative power of sparse representations for classification (SRC). Recently, it has been shown that the use of \emph{class-specific} spike-and-slab priors in conjunction with the class-specific dictionaries from SRC is particularly effective in low training scenarios. As a logical extension, we build on this framework for multitask scenarios, wherein multiple representations of the same physical phenomena are available. We experimentally demonstrate the benefits of mining joint information from differ…
In this paper, we examine the problem of real-world image deblurring and take into account two key factors for improving the performance of the deep image deblurring model, namely, training data synthesis and network architecture design. Deblurring models trained on existing synthetic datasets perform poorly on real blurry images due to domain shift. To reduce the domain gap between synthetic and real domains, we propose a novel realistic blur synthesis pipeline to simulate the camera imaging process. As a result of our proposed synthesis method, existing deblurring models could be made more r…
National University of Singapore（新加坡国立大学）；Singapore Institute for Neurotechnology（新加坡神经技术研究所）
In this paper, we address the challenging problem of action recognition, using event-based cameras. To recognise most gestural actions, often higher temporal precision is required for sampling visual information. Actions are defined by motion, and therefore, when using event-based cameras it is often unnecessary to re-sample the entire scene. Neuromorphic, event-based cameras have presented an alternative to visual information acquisition by asynchronously time-encoding pixel intensity changes, through temporally precise spikes (10 micro-second resolution), making them well equipped for action…
Learning to Robustly Reconstruct Low-light Dynamic Scenes from Spike Streams
School of Computer Science, Peking University（北京大学计算机学院）；Beijing Academy of Artificial Intelligence（北京人工智能研究院）；College of Future Technology, Peking University（北京大学未来技术学院）
ReImagine: Rethinking Controllable High-Quality Human Video Generation via Image-First Synthesis
School of Science and Engineering, The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）科学与工程学院）；Future Network of Intelligence Institute, CUHK-Shenzhen, China（CUHK-深圳智能网络研究院）
We present new theoretical foundations for unsupervised Spike-Timing-Dependent Plasticity (STDP) learning in spiking neural networks (SNNs). In contrast to empirical parameter search used in most previous works, we provide novel theoretical grounds for SNN and STDP parameter tuning which considerably reduces design time. Using our generic framework, we propose a class of global, action-based and convolutional SNN-STDP architectures for learning spatio-temporal features from event-based cameras. We assess our methods on the N-MNIST, the CIFAR10-DVS and the IBM DVS128 Gesture datasets, all acqui…
University of Chinese Academy of Sciences（中国科学院大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；CAS Center for Excellence in Brain Science and Intelligence Technology（中国科学院脑科学与智能技术卓越创新中心）
Pedestrian attribute recognition aims to assign multiple attributes to one pedestrian image captured by a video surveillance camera. Although numerous methods are proposed and make tremendous progress, we argue that it is time to step back and analyze the status quo of the area. We review and rethink the recent progress from three perspectives. First, given that there is no explicit and complete definition of pedestrian attribute recognition, we formally define and distinguish pedestrian attribute recognition from other similar tasks. Second, based on the proposed definition, we expose the lim…
