LoLDU: Low-Rank Adaptation via Lower-Diag-Upper Decomposition for Parameter-Efficient Fine-Tuning
University of Electronic Science and Technology of China（电子科技大学）；Center for Future Media（未来媒体中心）；School of Computer Science and Engineering（计算机科学与工程学院）
Yiming Shi, Jiwei Wei, Yujia Wu, Ran Ran, Chengwei Sun, Shiyuan He, Yang Yang
The rapid growth of model scale has necessitated substantial computational resources for fine-tuning. Existing approach such as Low-Rank Adaptation (LoRA) has sought to address the problem of handling the large updated parameters in full fine-tuning. However, LoRA utilize random initialization and optimization of low-rank matrices to approximate updated weights, which can result in suboptimal convergence and an accuracy gap compared to full fine-tuning. To address these issues, we propose LoLDU, a Parameter-Efficient Fine-Tuning (PEFT) approach that significantly reduces trainable parameters by 2600 times compared to regular PEFT methods while maintaining comparable performance. LoLDU leverages Lower-Diag-Upper Decomposition (LDU) to initialize low-rank matrices for faster convergence and orthogonality. We focus on optimizing the diagonal matrix for scaling transformations. To the best of our knowledge, LoLDU has the fewest parameters among all PEFT approaches. We conducted extensive experiments across 4 instruction-following datasets, 6 natural language understanding (NLU) datasets, 8 image classification datasets, and image generation datasets with multiple model types (LLaMA2, RoBERTa, ViT, and Stable Diffusion), providing a comprehensive and detailed analysis. Our open-source code can be accessed at \href{https://github.com/SKDDJ/LoLDU}{https://github.com/SKDDJ/LoLDU}.
ADAptation: Reconstruction-based Unsupervised Active Learning for Breast Ultrasound Diagnosis
Faculty of Applied Sciences, Macao Polytechnic University, Macau, China（澳门理工大学应用科学学院）；Medical Ultrasound Image Computing (MUSIC) Lab, School of Biomedical Engineering, Medical School, Shenzhen University, Shenzhen, China（深圳大学生物医学工程学院）；Netherlands Cancer Institute, Amsterdam, Netherlands（荷兰癌症研究所）；Department of Ultrasound, Peking University Third Hospital, Beijing, China（北京第三人民医院超声科）
MeLo: Low-rank Adaptation is Better than Fine-tuning for Medical Image Diagnosis
ShanghaiTech University（上海科技大学）；Shanghai Jiao Tong University（上海交通大学）；Shanghai United Imaging Intelligence Co., Ltd.（上海联影智能医疗科技有限公司）；Shanghai Clinical Research and Trial Center（上海临床研究与试验中心）
Fine-Tuning Cycle-GAN for Domain Adaptation of MRI Images
Department of Diagnostics and Intervention, and Biomedical Engineering, Umea University（乌梅大学诊断与介入系及生物医学工程系）；Department of Computer Science and Information Engineering, National Taipei University of Technology（国立台北科技大学计算机科学与信息工程系）；IRC for Bio Systems and Machines, King Fahd University of Petroleum and Minerals（国王法赫德石油与矿物大学生物系统与机器研究院）
TAPE: A two-stage parameter-efficient adaptation framework for foundation models in OCT-OCTA analysis
National Key Laboratory of Intelligent Tracking and Forecasting for Infectious Diseases（传染病智能追踪与预测国家重点实验室）；Tianjin Key Laboratory of Intelligent Robotic（天津市智能机器人重点实验室）；Institute of Robotics and Automatic Information System, Nankai University（南开大学机器人与自动化信息系统研究所）；Institute of Intelligence Technology and Robotic Systems（智能技术与机器人系统研究所）；Shenzhen Research Institute of Nankai University（南开大学深圳研究院）
Text-guided Foundation Model Adaptation for Long-Tailed Medical Image Classification
Southern University of Science and Technology（南方科技大学）；The University of Hong Kong（香港大学）；University of British Columbia（不列颠哥伦比亚大学）；Jiaxing Research Institute, Southern University of Science and Technology（南方科技大学嘉兴研究院）
GLT-PEFT: Gated Lie-Tucker Parameter-Efficient Fine-Tuning for Alzheimer's Disease Diagnosis with Hippocampal Segmentation Pretraining
Department of Mathematics, Hangzhou Dianzi University（杭州电子科技大学数学系）；School of Mathematics, Physics and Information, Shaoxing University（绍兴大学数学与物理学院）；Department of Mathematics, Zhejiang University of Science and Technology（浙江科技学院数学系）
Efficient Adaptation of Pre-trained Vision Transformer via Householder Transformation
College of Information and Control Engineering, Xi’an University of Architecture and Technology（西安建筑科技大学信息与控制工程学院）；School of Computer Science, Northwestern Polytechnical University（西北工业大学计算机学院）；School of Computer Science and Engineering, University of Electronic Science and Technology of China（电子科技大学计算机科学与工程学院）；School of Computer Science and Technology, Tongji University（同济大学计算机科学与技术学院）
