使用多模态脉冲神经网络进行数字识别
Digit Recognition using Multimodal Spiking Neural Networks
AI总结：
本文研究多模态脉冲神经网络在数字识别中的优势，融合视觉（N-MNIST）和听觉（SHD）事件数据，发现多模态SNN优于单模态，且融合深度不敏感，最终实现98.43%准确率。
摘要：
脉冲神经网络（SNNs）是第三代神经网络，其设计受生物启发，以模拟大脑中信号交换的方式处理数据。在计算机视觉领域，SNNs 获得了广泛关注，这在很大程度上得益于事件型传感器的可用性，这些传感器能够根据场景辐射度的变化产生空间分辨的脉冲序列。由于 SNNs 具有神经形态特性，它们被用于处理基于事件的数据。本文所提出的工作研究了在分类任务中融合多种感官输入的神经形态优势。具体而言，我们研究了 SNN 在数字分类中的性能，通过输入一个视觉模态分支（Neuromorphic-MNIST [N-MNIST]）和一个听觉模态分支（Spiking Heidelberg Digits [SHD]），这些数据集使用事件型传感器生成一系列时间相关事件。实验观察到，多模态 SNNs 的性能优于单模态视觉和单模态听觉 SNNs。此外，还观察到感官融合过程对视觉和听觉分支融合的深度不敏感。本文使用一个在较深层次拼接视觉和听觉分支的多模态 SNN，在 N-MNIST 和 SHD 组合数据集上实现了 98.43% 的准确率。
英文摘要：
Spiking neural networks (SNNs) are the third generation of neural networks that are biologically inspired to process data in a fashion that emulates the exchange of signals in the brain. Within the Computer Vision community SNNs have garnered significant attention due in large part to the availability of event-based sensors that produce a spatially resolved spike train in response to changes in scene radiance. SNNs are used to process event-based data due to their neuromorphic nature. The proposed work examines the neuromorphic advantage of fusing multiple sensory inputs in classification tasks. Specifically we study the performance of a SNN in digit classification by passing in a visual modality branch (Neuromorphic-MNIST [N-MNIST]) and an auditory modality branch (Spiking Heidelberg Digits [SHD]) from datasets that were created using event-based sensors to generate a series of time-dependent events. It is observed that multi-modal SNNs outperform unimodal visual and unimodal auditory SNNs. Furthermore, it is observed that the process of sensory fusion is insensitive to the depth at which the visual and auditory branches are combined. This work achieves a 98.43% accuracy on the combined N-MNIST and SHD dataset using a multimodal SNN that concatenates the visual and auditory branches at a late depth.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
