StyleGAN2 was demonstrated to be a powerful image generation engine that supports semantic editing. However, in order to manipulate a real-world image, one first needs to be able to retrieve its corresponding latent representation in StyleGAN's latent space that is decoded to an image as close as possible to the desired image. For many real-world images, a latent representation does not exist, which necessitates the tuning of the generator network. We present a per-image optimization method that tunes a StyleGAN2 generator such that it achieves a local edit to the generator's weights, resulting in almost perfect inversion, while still allowing image editing, by keeping the rest of the mapping between an input latent representation tensor and an output image relatively intact. The method is based on a one-shot training of a set of shallow update networks (aka. Gradient Modification Modules) that modify the layers of the generator. After training the Gradient Modification Modules, a modified generator is obtained by a single application of these networks to the original parameters, and the previous editing capabilities of the generator are maintained. Our experiments show a sizable gap in performance over the current state of the art in this very active domain. Our code is available at \url{https://github.com/sheffier/gani}.
Exploring User-level Gradient Inversion with a Diffusion Prior
Vanderbilt University(范德堡大学);University of Wisconsin-Madison(威斯康星大学麦迪逊分校);Mitsubishi Electric Research Laboratories(三菱电机研发实验室);Vanderbilt University Medical Center(范德堡大学医学中心)
Enhancing Gradient Inversion Attacks in Federated Learning via Hierarchical Feature Optimization
Tsinghua Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院);School of Computer Science and Technology, Harbin Institute of Technology(哈尔滨工业大学计算机科学与技术学院);Department of Computer Science and Technology, Tsinghua University(清华大学计算机科学与技术系)
GIFD: A Generative Gradient Inversion Method with Feature Domain Optimization
Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳));Tsinghua Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院);Peng Cheng Laboratory(鹏城实验室)
CGI-DM: Digital Copyright Authentication for Diffusion Models via Contrasting Gradient Inversion
Shanghai Jiao Tong University(上海交通大学);Queen’s University Belfast(贝尔法斯特女王大学);University of Southern California(南加州大学);Louisiana State University(路易斯安那州立大学)
GI-NAS: Boosting Gradient Inversion Attacks Through Adaptive Neural Architecture Search
Tsinghua Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院);School of Computer Science and Technology, Harbin Institute of Technology(哈尔滨工业大学计算机科学与技术学院);School of Computer Science and Engineering, Sun Yat-sen University(中山大学计算机科学与工程学院);Shenzhen ShenNong Information Technology Co., Ltd.(深圳神农信息技术有限公司);Department of Computer Science and Technology, Tsinghua University(清华大学计算机科学与技术系)
Towards Eliminating Hard Label Constraints in Gradient Inversion Attacks
School of Artificial Intelligence, University of Chinese Academy of Sciences (UCAS)(中国科学院大学人工智能学院);Institute of Automation, Chinese Academy of Sciences (CASIA)(中国科学院自动化研究所)
Get your Foes Fooled: Proximal Gradient Split Learning for Defense against Model Inversion Attacks on IoMT data
Tech University of Korea(韩国科技大学);University of Sindh(信德大学);Munster Technological University(明斯特理工大学);University of Johannesburg(约翰内斯堡大学);Sheffield Hallam University(谢菲尔德哈勒姆大学);Sungkyunkwan University(成均馆大学)
Shadow defense against gradient inversion attack in federated learning
Bioengineering Department, Imperial College London, London W12 0BZ, UK;School of Computer Engineering;Science, Shanghai university, Shanghai 200444, China;Bioengineering Department;Imperial-X, Imperial College London, London W12 7SL, UK;Lung Institute, Imperial College London, London SW7 2AZ, UK;Cardiovascular Research Centre, Royal Brompton Hospital, London SW3 6NP, UK;School of Biomedical Engineering \& Imaging Sciences, King's College London, London WC2R 2LS, UK