基于几何对齐的实时数据高效肖像风格化
Realtime Data-Efficient Portrait Stylization Based On Geometric Alignment
AI总结：
本文提出基于几何对齐的轻量级肖像风格化方法，通过整合可微TPS模块提升训练效率，实现2倍数据效率和100倍计算复杂度降低，支持移动设备实时推理。
摘要：
肖像风格化旨在通过风格示例赋予肖像照片生动的艺术效果。尽管有庞大的训练数据集和大网络权重，现有方法难以维持几何一致性并实现满意的风格化效果，因为面部照片和风格化图像之间的面部特征分布存在差异，限制了在稀有风格和移动设备上的应用。为此，我们提出在肖像和风格样本之间建立有意义的几何关联，通过对齐相应的面部特征来简化风格化过程。具体来说，我们将可微的薄板样条（TPS）模块整合到端到端的生成对抗网络（GAN）框架中，以提高训练效率并促进面部身份的一致性。通过利用面部的内在结构信息，例如面部关键点，TPS模块可以在像素和特征空间中在全局和局部尺度上建立两个领域的几何对齐，从而克服上述挑战。在一系列肖像风格化任务上的定量和定性比较表明，我们的模型不仅在保真度和风格一致性方面优于现有模型，还实现了2倍的数据效率提升和100倍的计算复杂度降低，使我们的轻量模型能够在移动设备上以512*512分辨率实现实时推理（30 FPS）。
英文摘要：
Portrait Stylization aims to imbue portrait photos with vivid artistic effects drawn from style examples. Despite the availability of enormous training datasets and large network weights, existing methods struggle to maintain geometric consistency and achieve satisfactory stylization effects due to the disparity in facial feature distributions between facial photographs and stylized images, limiting the application on rare styles and mobile devices. To alleviate this, we propose to establish meaningful geometric correlations between portraits and style samples to simplify the stylization by aligning corresponding facial characteristics. Specifically, we integrate differentiable Thin-Plate-Spline (TPS) modules into an end-to-end Generative Adversarial Network (GAN) framework to improve the training efficiency and promote the consistency of facial identities. By leveraging inherent structural information of faces, e.g., facial landmarks, TPS module can establish geometric alignments between the two domains, at global and local scales, both in pixel and feature spaces, thereby overcoming the aforementioned challenges. Quantitative and qualitative comparisons on a range of portrait stylization tasks demonstrate that our models not only outperforms existing models in terms of fidelity and stylistic consistency, but also achieves remarkable improvements in 2x training data efficiency and 100x less computational complexity, allowing our lightweight model to achieve real-time inference (30 FPS) at 512*512 resolution on mobile devices.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
