Face recognition systems store face templates for efficient matching. Once leaked, these templates pose a threat: inverting them can yield photorealistic surrogates that compromise privacy and enable impersonation. Although existing research has achieved relatively realistic face template inversion, the reconstructed facial images exhibit over-smoothed facial-part attributes (eyes, nose, mouth) and limited transferability. To address this problem, we present CLIP-FTI, a CLIP-driven fine-grained attribute conditioning framework for face template inversion. Our core idea is to use the CLIP model to obtain the semantic embeddings of facial features, in order to realize the reconstruction of specific facial feature attributes. Specifically, facial feature attribute embeddings extracted from CLIP are fused with the leaked template via a cross-modal feature interaction network and projected into the intermediate latent space of a pretrained StyleGAN. The StyleGAN generator then synthesizes face images with the same identity as the templates but with more fine-grained facial feature attributes. Experiments across multiple face recognition backbones and datasets show that our reconstructions (i) achieve higher identification accuracy and attribute similarity, (ii) recover sharper component-level attribute semantics, and (iii) improve cross-model attack transferability compared to prior reconstruction attacks. To the best of our knowledge, ours is the first method to use additional information besides the face template attack to realize face template inversion and obtains SOTA results.
The application of zero-shot learning in computer vision has been revolutionized by the use of image-text matching models. The most notable example, CLIP, has been widely used for both zero-shot classification and guiding generative models with a text prompt. However, the zero-shot use of CLIP is unstable with respect to the phrasing of the input text, making it necessary to carefully engineer the prompts used. We find that this instability stems from a selective similarity score, which is based only on a subset of the semantically meaningful input tokens. To mitigate it, we present a novel ex…
Image-to-Image Translation with Diffusion Transformers and CLIP-Based Image Conditioning
Department of Mechanical and Aerospace Engineering（机械与航空航天工程系）；University of Houston（休斯顿大学）；School of Engineering（工程学院）；Santa Clara University（圣克拉拉大学）；School of Electrical and Computer Engineering（电气与计算机工程学院）；Cornell University（康奈尔大学）；Department of Electrical and Computer Engineering（电气与计算机工程系）；Northeastern University（东北大学）
Parameter-Efficient Vision-Language Adaptation with Continuous Metadata Conditioning for Animal Re-Identification
Department of Computer Science, University of Verona（维罗纳大学计算机科学系）；Institute of Marine Research（海洋研究所）；University of Agder, Centre for Coastal Research（阿格德大学海岸研究中心）
City University of Hong Kong（香港城市大学）；Snap Inc.（Snap公司）；USC Institute for Creative Technologies（南加州大学创意技术研究所）；Microsoft Cloud AI（微软云人工智能部门）
We present CLIP-NeRF, a multi-modal 3D object manipulation method for neural radiance fields (NeRF). By leveraging the joint language-image embedding space of the recent Contrastive Language-Image Pre-Training (CLIP) model, we propose a unified framework that allows manipulating NeRF in a user-friendly way, using either a short text prompt or an exemplar image. Specifically, to combine the novel view synthesis capability of NeRF and the controllable manipulation ability of latent representations from generative models, we introduce a disentangled conditional NeRF architecture that allows indiv…
Guiding Diffusion-based Reconstruction with Contrastive Signals for Balanced Visual Representation
State Key Laboratory of AI Safety, Institute of Computing Technology, CAS（人工智能安全国家重点实验室，计算技术研究所，中国科学院）；School of Computer Science and Tech., University of Chinese Academy of Sciences（中国科学院大学计算机科学与技术学院）；Beijing Academy of Artificial Intelligence（北京人工智能研究院）；Institute of Information Engineering, CAS（信息工程研究所，中国科学院）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络安全学院）；School of Computer Science and Technology, Beijing Institute of Technology（北京理工大学计算机科学与技术学院）
