使用GANs自然化神经形态视觉事件流
Naturalizing Neuromorphic Vision Event Streams Using GANs
摘要：
动态视觉传感器能够在资源受限的环境中以高时间分辨率操作，尽管以捕获静态内容为代价。事件流的稀疏性质使得能够实现高效的下游处理任务，因为它们适合于功率高效的脉冲神经网络。与神经形态视觉相关的挑战之一是缺乏事件流的可解释性。虽然大多数应用用例不打算通过分类网络以外的任何东西来视觉解释事件流，但在传统高速CMOS传感器无法进入的空间中集成这些传感器的机会已经丧失。例如，生物侵入性传感器（如内窥镜）必须符合严格的功率预算，这不允许图像集成的MHz速度。虽然动态视觉传感可以填补这一空白，但解释挑战仍然存在，并将降低临床诊断的信心。生成对抗网络的使用为克服和补偿视觉芯片的空间分辨率差和缺乏可解释性提供了一种可能的解决方案。在本文中，我们有条不紊地应用Pix 2 Pix网络来自然化来自尖峰转换的CIFAR-10和Linnaeus 5数据集的事件流。通过对自然化事件流进行图像分类，对网络的质量进行基准测试，其收敛到等效原始图像的2.81%以内，并且对于CIFAR-10和Linnaeus 5数据集，与未处理事件流相比，相关改进为13.19%。
英文摘要：
Dynamic vision sensors are able to operate at high temporal resolutions within resource constrained environments, though at the expense of capturing static content. The sparse nature of event streams enables efficient downstream processing tasks as they are suited for power-efficient spiking neural networks. One of the challenges associated with neuromorphic vision is the lack of interpretability of event streams. While most application use-cases do not intend for the event stream to be visually interpreted by anything other than a classification network, there is a lost opportunity to integrating these sensors in spaces that conventional high-speed CMOS sensors cannot go. For example, biologically invasive sensors such as endoscopes must fit within stringent power budgets, which do not allow MHz-speeds of image integration. While dynamic vision sensing can fill this void, the interpretation challenge remains and will degrade confidence in clinical diagnostics. The use of generative adversarial networks presents a possible solution to overcoming and compensating for a vision chip's poor spatial resolution and lack of interpretability. In this paper, we methodically apply the Pix2Pix network to naturalize the event stream from spike-converted CIFAR-10 and Linnaeus 5 datasets. The quality of the network is benchmarked by performing image classification of naturalized event streams, which converges to within 2.81% of equivalent raw images, and an associated improvement over unprocessed event streams by 13.19% for the CIFAR-10 and Linnaeus 5 datasets.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
