通过解耦几何-光度数据合成学习双摄像头平滑变焦
Learning Dual-Camera Smooth Zoom via Decoupled Geometric-Photometric Data Synthesis
AI总结:
本文针对双摄像头变焦的不连续问题,提出ZoomGS v2数据合成框架与ZoomFI专用VFI网络,经实验验证可提升VFI模型性能并达到最优结果。
摘要:
在智能手机双摄像头之间变焦时,几何视角和色彩还原都会出现明显的不连续问题,不可避免地降低用户的视觉体验。为解决该问题,本文提出双摄像头平滑变焦(DCSZ)任务,旨在合成连续的中间帧以实现流畅变焦。然而,直接将现有视频帧插值(VFI)模型应用于该任务会受到显著的领域差距以及难以获取真实世界连续真值的严重阻碍。为此,本文提出一种生成高保真DCSZ训练序列的新型数据合成框架,命名为ZoomGS v2。该框架明确地从3D场景表示建模中解耦几何和色彩变换,从而能够从易于获取的单摄像头3D场景中生成大量数据。此外,本文开发了专门针对DCSZ任务的VFI网络ZoomFI,其利用互补双向扭曲实现全面的双摄像头信息融合。在使用两部手机采集的测试数据集上进行的大量实验表明,使用构建的DCSZ数据进行训练可显著提升各类VFI模型的性能。此外,ZoomFI在定量指标和视觉质量上均达到了最先进的结果。数据集、代码和预训练模型可在该https网址获取。
英文摘要:
When zooming between dual cameras on smartphones, prominent discontinuities in both geometric perspective and color rendition inevitably degrade the visual experience of users. To address this issue, we introduce the dual-camera smooth zoom (DCSZ) task, aiming to synthesize continuous intermediate frames for fluid zooming. However, directly applying existing video frame interpolation (VFI) models to this task is severely hindered by substantial domain gaps and the practical intractability of capturing real-world continuous ground-truth. To this end, we propose a novel data synthesis framework to generate high-fidelity DCSZ training sequences, named ZoomGS v2. It explicitly decouples geometric and color transformations from 3D scene representation modeling, thereby enabling extensive data generation from easily accessible single-camera 3D scenes. Furthermore, we develop ZoomFI, a specialized VFI network tailored to the DCSZ task, which leverages complementary bidirectional warping for comprehensive dual-camera information fusion. Extensive experiments on test datasets captured using two mobile phones demonstrate that training with the constructed DCSZ data significantly improves the performance of various VFI models. Moreover, ZoomFI achieves state-of-the-art results in both quantitative metrics and visual quality. The datasets, code, and pretrained models are available at https://hitjunjief.github.io/DCSZ-v2
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。