Generative models have achieved remarkable success in image, video, and text domains. Inspired by this, researchers have explored utilizing generative models to generate neural network parameters. However, these efforts have been limited by the parameter size and the practicality of generating high-performance parameters. In this paper, we propose COND P-DIFF, a novel approach that demonstrates the feasibility of controllable high-performance parameter generation, particularly for LoRA (Low-Rank Adaptation) weights, during the fine-tuning process. Specifically, we employ an autoencoder to extract efficient latent representations for parameters. We then train a conditional latent diffusion model to synthesize high-performing model parameters from random noise based on specific task conditions. Experimental results in both computer vision and natural language processing domains consistently demonstrate that COND P-DIFF can generate high-performance parameters conditioned on the given task. Moreover, we observe that the parameter distribution generated by COND P-DIFF exhibits differences compared to the distribution obtained through normal optimization methods, indicating a certain level of generalization capability. Our work paves the way for further exploration of condition-driven parameter generation, offering a promising direction for task-specific adaptation of neural networks.
Department of Computing/LSGI, The Hong Kong Polytechnic University, Hong Kong SAR(计算系/LSGI,香港理工大学,香港特别行政区);School of Artificial Intelligence, Xidian University, China(人工智能学院,西安电子科技大学,中国)
LoRA-Pro: Are Low-Rank Adapters Properly Optimized?
University of Science and Technology of China(中国科学技术大学);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Nanjing University(南京大学)
Enhancing LLMs for Graph Tasks via Graph-aware LoRA Generation
State Key Lab. of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所人工智能安全国家重点实验室);University of Chinese Academy of Sciences(中国科学院大学);DAMO Academy, Alibaba Group(阿里巴巴集团达摩院)
LoRA Fine-Tuning Without GPUs: A CPU-Efficient Meta-Generation Framework for LLMs
Department of Mathematics, McMaster University, Hamilton, Canada(麦斯特大学数学系);Vector Institute, Toronto, Canada(向量研究所);University of Oxford, Oxford, United Kingdom(牛津大学)
LoRA-One: One-Step Full Gradient Could Suffice for Fine-Tuning Large Language Models, Provably and Efficiently
Department of Statistics, University of Warwick, UK.(威斯敏斯特大学统计学系);Department of Computer Science, University of Warwick, UK.(威斯敏斯特大学计算机科学系);Centre for Discrete Mathematics and its Applications (DIMAP), University of Warwick, UK.(威斯敏斯特大学离散数学及其应用中心(DIMAP))
Few-shot multi-token DreamBooth with LoRa for style-consistent character generation
Institute of Smart Cities (ISC)(智能城市研究所);Department of Statistics, Computer Science and Mathematics(统计学、计算机科学与数学系);Public University of Navarre (UPNA)(纳瓦拉公共大学)
Advancing AI-Powered Medical Image Synthesis: Insights from MedVQA-GI Challenge Using CLIP, Fine-Tuned Stable Diffusion, and Dream-Booth + LoRA
Department of Computer Science, SCMNS School, Morgan State University, Baltimore, Maryland 21251, USA;Electrical \& Computer Engineering Dept., School of Engineering, Morgan State University, Baltimore, Maryland 21251, USA
A Unified Study of LoRA Variants: Taxonomy, Review, Codebase, and Empirical Evaluation
Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);University of Science and Technology of China(中国科学技术大学);Fudan University(复旦大学);The Chinese University of Hong Kong(香港中文大学);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所)