arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2404.04908cs.CV
收藏

手机双摄平滑变焦

Dual-Camera Smooth Zoom on Mobile Phones

Harbin Institute of Technology(哈尔滨工业大学)

Renlong Wu, Zhilu Zhang, Yu Yang, Wangmeng Zuo

AI总结:

针对手机双摄变焦时画面跳变的问题,提出双摄平滑变焦任务,构建基于ZoomGS的数据工厂生成合成数据以微调帧插值模型,显著提升了双摄平滑变焦效果。

摘要:

在手机双摄像头之间进行变焦时,预览画面会出现几何内容和图像色彩的明显跳变,不可避免地影响用户的变焦体验。本文提出了一项新任务——双摄平滑变焦(DCSZ),旨在实现流畅的变焦预览。帧插值(FI)技术是一种潜在的解决方案,但面临真值数据采集困难的问题。为解决这一难题,我们提出了一种数据工厂方案:通过搭建连续虚拟摄像头,对重建的场景3D模型进行渲染,从而生成DCSZ数据。具体而言,我们提出了一种新颖的双摄平滑变焦高斯泼溅(ZoomGS)方法,引入摄像头专属编码,为每个虚拟摄像头构建专属3D模型。借助所提出的数据工厂,我们构建了一个DCSZ合成数据集,并用它来微调帧插值模型。此外,我们还采集了无真值的真实世界双变焦图像用于评估。我们使用多种帧插值方法开展了大量实验,结果表明,经过微调的帧插值模型在DCSZ任务上的性能较原始模型有显著提升。数据集、代码和预训练模型将发布于https://github.com/ZcsrenlongZ/ZoomGS。

英文摘要:

When zooming between dual cameras on a mobile, noticeable jumps in geometric content and image color occur in the preview, inevitably affecting the user's zoom experience. In this work, we introduce a new task, ie, dual-camera smooth zoom (DCSZ) to achieve a smooth zoom preview. The frame interpolation (FI) technique is a potential solution but struggles with ground-truth collection. To address the issue, we suggest a data factory solution where continuous virtual cameras are assembled to generate DCSZ data by rendering reconstructed 3D models of the scene. In particular, we propose a novel dual-camera smooth zoom Gaussian Splatting (ZoomGS), where a camera-specific encoding is introduced to construct a specific 3D model for each virtual camera. With the proposed data factory, we construct a synthetic dataset for DCSZ, and we utilize it to fine-tune FI models. In addition, we collect real-world dual-zoom images without ground-truth for evaluation. Extensive experiments are conducted with multiple FI methods. The results show that the fine-tuned FI models achieve a significant performance improvement over the original ones on DCSZ task. The datasets, codes, and pre-trained models will are available at https://github.com/ZcsrenlongZ/ZoomGS.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑