With the emerging trend in generative models and convenient public access to diffusion models pre-trained on large datasets, users can fine-tune these models to generate images of personal faces or items in new contexts described by natural language. Parameter efficient fine-tuning (PEFT) such as Low Rank Adaptation (LoRA) has become the most common way to save memory and computation usage on the user end during fine-tuning. However, a natural question is whether the private images used for fine-tuning will be leaked to adversaries when sharing model weights. In this paper, we study the issue of privacy leakage of a fine-tuned diffusion model in a practical setting, where adversaries only have access to model weights, rather than prompts or images used for fine-tuning. We design and build a variational network autoencoder that takes model weights as input and outputs the reconstruction of private images. To improve the efficiency of training such an autoencoder, we propose a training paradigm with the help of timestep embedding. The results give a surprising answer to this research question: an adversary can generate images containing the same identities as the private images. Furthermore, we demonstrate that no existing defense method, including differential privacy-based methods, can preserve the privacy of private data used for fine-tuning a diffusion model without compromising the utility of a fine-tuned model.
Advancing AI-Powered Medical Image Synthesis: Insights from MedVQA-GI Challenge Using CLIP, Fine-Tuned Stable Diffusion, and Dream-Booth + LoRA
Department of Computer Science, SCMNS School, Morgan State University, Baltimore, Maryland 21251, USA；Electrical \& Computer Engineering Dept., School of Engineering, Morgan State University, Baltimore, Maryland 21251, USA
LoRA Diffusion: Zero-Shot LoRA Synthesis for Diffusion Model Personalization
Leonardo AI Research Lab（莱昂纳多人工智能研究实验室）；Lucid Simulations, Corp.（Lucid Simulations公司）；Precigenetics Inc（Precigenetics公司）；SimpleBerry Research Lab（SimpleBerry研究实验室）；University of California, Merced（加州大学默塞德分校）
Filtering Memorization from Parameter-Space in Diffusion Models
RIKEN AIP（理化学研究所Advanced Institute for Peripheral Research）；Science of Tokyo（东京科学大学）；University of California, Berkeley（加州大学伯克利分校）；Zhejiang University（浙江大学）
TimeStep Master: Asymmetrical Mixture of Timestep LoRA Experts for Versatile and Efficient Diffusion Models in Vision
Shanghai Jiao Tong University（上海交通大学）；Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；Shanghai AI Laboratory（上海人工智能实验室）；Zhejiang University（浙江大学）；WeChat, Tencent Inc（腾讯微信）
Task-Driven Subspace Decomposition for Knowledge Sharing and Isolation in LoRA-based Continual Learning
Department of XXX, University of YYY, Location, Country（XXX部门，YYY大学，地点，国家）；School of ZZZ, Institute of WWW, Location, Country（ZZZ学院，WWW研究所，地点，国家）；State Key Laboratory of Integrated Services Networks, School of Telecommunications Engineering, Xidian University, Xi'an, China（信息服务网络国家重点实验室，电信工程学院，西安电子科技大学，西安，中国）；School of Electronic Engineering, Xidian University, Xi'an, China（电子工程学院，西安电子科技大学，西安，中国）
Uncertainty quantification in fine-tuned LLMs using LoRA ensembles
Stockholm University Department of Physics（斯德哥尔摩大学物理系）；Department of Mathematical Sciences（数学科学系）；Chalmers university of technology & University of Gothenburg（楚德勒斯技术大学及哥德堡大学）；VERSES Research Lab（VERSES研究实验室）
An important paradigm of natural language processing consists of large-scale pre-training on general domain data and adaptation to particular tasks or domains. As we pre-train larger models, full fine-tuning, which retrains all model parameters, becomes less feasible. Using GPT-3 175B as an example -- deploying independent instances of fine-tuned models, each with 175B parameters, is prohibitively expensive. We propose Low-Rank Adaptation, or LoRA, which freezes the pre-trained model weights and injects trainable rank decomposition matrices into each layer of the Transformer architecture, grea…
How Relevance Emerges: Interpreting LoRA Fine-Tuning in Reranking LLMs
Center for Intelligent Information Retrieval, Manning College of Information and Computer Sciences, University of Massachusetts Amherst（智能信息检索中心，信息与计算机科学学院，马萨诸塞大学阿姆赫斯特分校）
HAFLQ: Heterogeneous Adaptive Federated LoRA Fine-tuned LLM with Quantization
Department of Engineering, King’s College London（伦敦国王学院工程系）；Advanced Technology Group, Ericsson Inc.（爱立信公司先进科技组）；Institute for Digital Communication, Friedrich-Alexander-Universität Erlangen-Nürnberg（埃尔兰根-纽伦堡弗里德里希-亚历山大大学数字通信研究所）
LoRALib: A Standardized Benchmark for Evaluating LoRA-MoE Methods
Zhejiang University of Technology（浙江工业大学）；The City University of New York（纽约城市大学）；Binjiang Institute of Artificial Intelligence, Zhejiang University of Technology（浙江工业大学滨江人工智能研究院）；A*STAR（新加坡科技研究局）
R-LoRA: Randomized Multi-Head LoRA for Efficient Multi-Task Learning
School of Artificial Intelligence, Jilin University（吉林大学人工智能学院）；Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, MOE, China（知识驱动人机智能工程研究中心，教育部，中国）；International Center of Future Science, Jilin University（未来科学国际中心，吉林大学）
Beyond LoRA: Is Sparsity-Induced Adaptation Better?
School of Data, Mathematical and Statistical Sciences, University of Central Florida, United States（中佛罗里达大学数据、数学与统计科学学院）；College of Computing, Mohammed VI Polytechnic University (UM6P), Morocco（穆罕默德六世理工大学计算机学院）；Department of Computer Science, University of Central Florida, United States（中佛罗里达大学计算机科学系）
