arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2210.05451cs.CVeess.IV
收藏

实现无ISP低功耗计算机视觉

Enabling ISP-less Low-Power Computer Vision

Gourav Datta, Zeyu Liu, Zihan Yin, Linyu Sun, Akhilesh R. Jaiswal, Peter A. Beerel

摘要:

为了在资源受限的低功率设备上部署当前的计算机视觉(CV)模型,最近的工作已经提出了传感器内和像素内计算方法,其试图部分/完全绕过图像信号处理器(ISP),并且通过对初始卷积神经网络(CNN)层中的激活图进行下采样来在图像传感器和CV处理单元之间产生显著的带宽减少。然而,由于与用于训练的ISP处理图像相比,由图像传感器捕获的原始图像的协方差的差异,对原始图像的直接推断降低了测试精度。此外,很难在原始图像上训练深度CV模型,因为大多数(如果不是全部)大型开源数据集由RGB图像组成。为了减轻这种担忧,我们建议反转ISP管道,它可以将任何数据集的RGB图像转换为原始图像,并在原始图像上进行模型训练。我们发布了COCO数据集的原始版本,这是通用高级视觉任务的大规模基准。对于无ISP的CV系统,与依赖传统ISP处理的RGB数据集的训练相比,对这些原始图像进行训练可使视觉唤醒作品(VWW)数据集的测试准确度提高7.1%。为了进一步提高无ISP CV模型的准确性并增加通过传感器内/像素内计算获得的能量和带宽益处,我们提出了一种节能形式的模拟像素内去马赛克,其可以与像素内CNN计算相结合。当对PASCALRAW数据集的真实传感器捕获的原始图像进行评估时,我们的方法导致mAP增加了8.1%。最后,我们证明了进一步增加20.5%的mAP通过使用一种新的应用程序的少样本学习与30杆,每个新的PASCALRAW数据集,构成3类。

英文摘要:

In order to deploy current computer vision (CV) models on resource-constrained low-power devices, recent works have proposed in-sensor and in-pixel computing approaches that try to partly/fully bypass the image signal processor (ISP) and yield significant bandwidth reduction between the image sensor and the CV processing unit by downsampling the activation maps in the initial convolutional neural network (CNN) layers. However, direct inference on the raw images degrades the test accuracy due to the difference in covariance of the raw images captured by the image sensors compared to the ISP-processed images used for training. Moreover, it is difficult to train deep CV models on raw images, because most (if not all) large-scale open-source datasets consist of RGB images. To mitigate this concern, we propose to invert the ISP pipeline, which can convert the RGB images of any dataset to its raw counterparts, and enable model training on raw images. We release the raw version of the COCO dataset, a large-scale benchmark for generic high-level vision tasks. For ISP-less CV systems, training on these raw images result in a 7.1% increase in test accuracy on the visual wake works (VWW) dataset compared to relying on training with traditional ISP-processed RGB datasets. To further improve the accuracy of ISP-less CV models and to increase the energy and bandwidth benefits obtained by in-sensor/in-pixel computing, we propose an energy-efficient form of analog in-pixel demosaicing that may be coupled with in-pixel CNN computations. When evaluated on raw images captured by real sensors from the PASCALRAW dataset, our approach results in a 8.1% increase in mAP. Lastly, we demonstrate a further 20.5% increase in mAP by using a novel application of few-shot learning with thirty shots each for the novel PASCALRAW dataset, constituting 3 classes.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑