arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2408.07530cs.CV
收藏

面向移动设备的实时视频压缩感知

Towards Real-time Video Compressive Sensing on Mobile Devices

Zhejiang University(浙江大学);Westlake University(西湖大学);University of Electronic Science and Technology of China(电子科技大学)

Miao Cao, Lishun Wang, Huan Wang, Guoqing Wang, Xin Yuan

AI总结:

针对移动设备难以实时运行视频SCI重建算法的问题,提出MobileSCI方法,通过U型2D卷积架构、高效特征混合模块与定制知识蒸馏,在iPhone 15上实现35 FPS的实时高质量重建。

摘要:

视频快照压缩成像(Video Snapshot Compressive Imaging, SCI)采用低速2D相机将高速场景捕获为快照压缩测量值,再通过重建算法恢复出高速视频帧。快速发展的移动设备与现有高性能视频SCI重建算法,促使我们为实际应用开发移动端重建方法。然而,由于推理过程复杂,以往的重建算法难以部署在移动设备上,更无法实现移动端实时重建。据我们所知,目前尚无专门为移动设备运行设计的视频SCI重建模型。针对这一问题,本文提出了一种高效的视频SCI重建方法,命名为MobileSCI,首次实现了在移动设备上的实时运行。具体而言,我们首先构建了一种基于U型2D卷积的架构,相比现有最优重建方法效率更高、更适配移动端。此外,我们提出一种基于通道拆分与混洗机制的高效特征混合模块,作为MobileSCI的新型瓶颈模块,以减轻计算负担。最后,我们采用定制化的知识蒸馏策略进一步提升重建质量。在模拟数据与真实数据上的大量实验结果表明,本文提出的MobileSCI在移动设备上兼具优异的重建质量与高效性。具体来说,在iPhone 15上,我们对尺寸为256×256×8的快照压缩测量值进行重建可达到实时性能(约35 FPS)。代码已开源至https://github.com/mcao92/MobileSCI。

英文摘要:

Video Snapshot Compressive Imaging (SCI) uses a low-speed 2D camera to capture high-speed scenes as snapshot compressed measurements, followed by a reconstruction algorithm to retrieve the high-speed video frames. The fast evolving mobile devices and existing high-performance video SCI reconstruction algorithms motivate us to develop mobile reconstruction methods for real-world applications. Yet, it is still challenging to deploy previous reconstruction algorithms on mobile devices due to the complex inference process, let alone real-time mobile reconstruction. To the best of our knowledge, there is no video SCI reconstruction model designed to run on the mobile devices. Towards this end, in this paper, we present an effective approach for video SCI reconstruction, dubbed MobileSCI, which can run at real-time speed on the mobile devices for the first time. Specifically, we first build a U-shaped 2D convolution-based architecture, which is much more efficient and mobile-friendly than previous state-of-the-art reconstruction methods. Besides, an efficient feature mixing block, based on the channel splitting and shuffling mechanisms, is introduced as a novel bottleneck block of our proposed MobileSCI to alleviate the computational burden. Finally, a customized knowledge distillation strategy is utilized to further improve the reconstruction quality. Extensive results on both simulated and real data show that our proposed MobileSCI can achieve superior reconstruction quality with high efficiency on the mobile devices. Particularly, we can reconstruct a 256 X 256 X 8 snapshot compressed measurement with real-time performance (about 35 FPS) on an iPhone 15. Code is available at https://github.com/mcao92/MobileSCI.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑