arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2310.12092cs.CV
收藏

HSTR-Net:基于参考的双摄像头视频超分辨率

HSTR-Net: Reference Based Video Super-resolution with Dual Cameras

University of Arizona(亚利桑那大学);Ozyegin University(奥兹耶金大学);Istanbul Medipol University(伊斯坦布尔梅迪波尔大学)

H. Umut Suluhan, Abdullah Enes Doruk, Hasan F. Ates, Bahadir K. Gunturk

AI总结:

本文提出双摄像头 HSTR-Net,通过融合 HSLF 与 LSHF 视频并结合光流和注意力机制进行 RefSR,在指标和无人机空中监测实时性上均表现优异。

摘要:

高时空分辨率(HSTR)视频录制在增强各类需要精细细节信息的成像任务方面发挥着关键作用。最先进的相机能够同时提供所需的高帧率和高空间分辨率,但成本高昂。为缓解这一问题,本文提出了一种利用基于参考的超分辨率(RefSR)生成 HSTR 视频的双摄像头系统。其中一个摄像头捕获高空间分辨率、低帧率(HSLF)视频,另一个同时针对同一场景捕获低空间分辨率、高帧率(LSHF)视频。本文提出了一种新颖的深度学习架构,用于融合 HSLF 和 LSHF 视频流并合成 HSTR 视频帧。所提模型结合了光流估计和(通道维度与空间维度)注意力机制,以捕捉两路视频流帧之间的精细运动和复杂依赖关系。仿真表明,所提模型在 PSNR 和 SSIM 指标上相较现有基于参考的 SR 技术取得了显著提升。当部署在配备双摄像头且功耗受限的无人机上时,该方法还展现出足以用于空中监测的每秒帧数(FPS)。

英文摘要:

High-spatio-temporal resolution (HSTR) video recording plays a crucial role in enhancing various imagery tasks that require fine-detailed information. State-of-the-art cameras provide this required high frame-rate and high spatial resolution together, albeit at a high cost. To alleviate this issue, this paper proposes a dual camera system for the generation of HSTR video using reference-based super-resolution (RefSR). One camera captures high spatial resolution low frame rate (HSLF) video while the other captures low spatial resolution high frame rate (LSHF) video simultaneously for the same scene. A novel deep learning architecture is proposed to fuse HSLF and LSHF video feeds and synthesize HSTR video frames. The proposed model combines optical flow estimation and (channel-wise and spatial) attention mechanisms to capture the fine motion and complex dependencies between frames of the two video feeds. Simulations show that the proposed model provides significant improvement over existing reference-based SR techniques in terms of PSNR and SSIM metrics. The method also exhibits sufficient frames per second (FPS) for aerial monitoring when deployed on a power-constrained drone equipped with dual cameras.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑