CLIP4Sketch：通过使用扩散模型进行数据集增强来提升素描与大头照匹配
CLIP4Sketch: Enhancing Sketch to Mugshot Matching through Dataset Augmentation using Diffusion Models
AI总结：
提出 CLIP4Sketch 方法，利用扩散模型结合 CLIP 和 Adaface 嵌入生成受控素描数据集，显著提升了法医素描与大头照匹配的准确率，并优于 GAN 方法。
摘要：
法医素描与大头照匹配是人脸识别中的一项具有挑战性的任务，主要障碍在于带标注的法医素描稀缺，以及素描与照片之间的模态差异。为解决此问题，我们提出了 CLIP4Sketch，一种利用扩散模型生成大量且多样化的素描图像集的新方法，有助于提升人脸识别系统在素描与大头照匹配中的性能。我们的方法利用去噪扩散概率模型（DDPMs）生成素描，并对身份和风格进行显式控制。我们将参考大头照的 CLIP 和 Adaface 嵌入，连同风格的文本描述，作为扩散模型的条件。我们通过生成对应于大头照的综合素描数据集，并在我们的合成数据上训练人脸识别模型，展示了该方法的有效性。结果表明，与在现有有限数量的真实人脸素描数据上训练相比，素描与大头照匹配的准确率有显著提高，验证了扩散模型在提升跨模态人脸识别系统性能方面的潜力。我们还将我们的数据集与使用基于 GAN 的方法生成的数据集进行了比较，以显示其优越性。
英文摘要：
Forensic sketch-to-mugshot matching is a challenging task in face recognition, primarily hindered by the scarcity of annotated forensic sketches and the modality gap between sketches and photographs. To address this, we propose CLIP4Sketch, a novel approach that leverages diffusion models to generate a large and diverse set of sketch images, which helps in enhancing the performance of face recognition systems in sketch-to-mugshot matching. Our method utilizes Denoising Diffusion Probabilistic Models (DDPMs) to generate sketches with explicit control over identity and style. We combine CLIP and Adaface embeddings of a reference mugshot, along with textual descriptions of style, as the conditions to the diffusion model. We demonstrate the efficacy of our approach by generating a comprehensive dataset of sketches corresponding to mugshots and training a face recognition model on our synthetic data. Our results show significant improvements in sketch-to-mugshot matching accuracy over training on an existing, limited amount of real face sketch data, validating the potential of diffusion models in enhancing the performance of face recognition systems across modalities. We also compare our dataset with datasets generated using GAN-based methods to show its superiority.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
