Point-PEFT: Parameter-Efficient Fine-Tuning for 3D Pre-trained Models
Shanghai AI Laboratory（上海人工智能实验室）；Northwestern Polytechnical University（西北工业大学）
Yiwen Tang, Ray Zhang, Zoey Guo, Dong Wang, Zhigang Wang, Bin Zhao, Xuelong Li
备注：
The specialized PEFT framework for 3D pre-trained models, which achieves competitive performance to full fine-tuning, and significantly reduces the computational resources. Project page: https://github.com/Ivan-Tang-3D/Point-PEFT
The popularity of pre-trained large models has revolutionized downstream tasks across diverse fields, such as language, vision, and multi-modality. To minimize the adaption cost for downstream tasks, many Parameter-Efficient Fine-Tuning (PEFT) techniques are proposed for language and 2D image pre-trained models. However, the specialized PEFT method for 3D pre-trained models is still under-explored. To this end, we introduce Point-PEFT, a novel framework for adapting point cloud pre-trained models with minimal learnable parameters. Specifically, for a pre-trained 3D model, we freeze most of its parameters, and only tune the newly added PEFT modules on downstream tasks, which consist of a Point-prior Prompt and a Geometry-aware Adapter. The Point-prior Prompt adopts a set of learnable prompt tokens, for which we propose to construct a memory bank with domain-specific knowledge, and utilize a parameter-free attention to enhance the prompt tokens. The Geometry-aware Adapter aims to aggregate point cloud features within spatial neighborhoods to capture fine-grained geometric information through local interactions. Extensive experiments indicate that our Point-PEFT can achieve better performance than the full fine-tuning on various downstream tasks, while using only 5% of the trainable parameters, demonstrating the efficiency and effectiveness of our approach. Code is released at https://github.com/Ivan-Tang-3D/Point-PEFT.
Standard fine-tuning of large pre-trained language models (PLMs) for downstream tasks requires updating hundreds of millions to billions of parameters, and storing a large copy of the PLM weights for every task resulting in increased cost for storing, sharing and serving the models. To address this, parameter-efficient fine-tuning (PEFT) techniques were introduced where small trainable components are injected in the PLM and updated during fine-tuning. We propose AdaMix as a general PEFT method that tunes a mixture of adaptation modules -- given the underlying PEFT method of choice -- introduce…
Standard fine-tuning of large pre-trained language models (PLMs) for downstream tasks requires updating hundreds of millions to billions of parameters, and storing a large copy of the PLM weights for every task resulting in increased cost for storing, sharing and serving the models. To address this, parameter-efficient fine-tuning (PEFT) techniques were introduced where small trainable components are injected in the PLM and updated during fine-tuning. We propose AdaMix as a general PEFT method that tunes a mixture of adaptation modules -- given the underlying PEFT method of choice -- introduce…
Parameter-Efficient Fine-Tuning for Pre-Trained Vision Models: A Survey and Benchmark
Nanjing University（南京大学）；Shanghai Innovation Institute（上海创新研究院）；Shanghai Jiao Tong University（上海交通大学）；University of Bristol（布里斯托大学）；Shandong University（山东大学）；University of Sydney（悉尼大学）；University of Minnesota Twin Cities（明尼苏达大学双城分校）；The Pennsylvania State University（宾夕法尼亚州立大学）；Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（深圳先进技术研究院，中国科学院）；University of California at Merced（加州大学默塞德分校）
Optimizing the Phi-2 Small Language Model for Real-time Chatbot Applications Using Parameter-Efficient Fine-Tuning (PEFT) with QLoRA Quantization
Beacom College of Computer and Cyber Sciences, Dakota State University（达科他州立大学比科姆计算机与网络科学学院）；Mathematics and Computer Science, Augustana College（奥古斯塔纳学院数学与计算机科学系）
PEFT-MuTS: A Multivariate Parameter-Efficient Fine-Tuning Framework for Remaining Useful Life Prediction based on Cross-domain Time Series Representation Model
School of Intelligence Science and Technology, University of Science and Technology Beijing（智能科学与技术学院，北京科技大学）；School of Automation and Electrical Engineering, University of Science and Technology Beijing（自动化与电气工程学院，北京科技大学）；Key Laboratory of Knowledge Automation for Industrial Processes of the Ministry of Education, University of Science and Technology Beijing（教育部知识自动化工业过程重点实验室，北京科技大学）；Rocket Force University of Engineering（火箭军工程大学）；Institute of Embodied Intelligence, Anhui University（embodied intelligence 院，安徽大学）
Efficient Fine-Tuning Methods for Portuguese Question Answering: A Comparative Study of PEFT on BERTimbau and Exploratory Evaluation of Generative LLMs
Institute of Science and Technology（科学技术研究院）；Federal University of São Paulo (UNIFESP)（圣保罗联邦大学）
Bilevel ZOFO: Efficient LLM Fine-Tuning and Meta-Training
Department of Computer Science University of Maryland - College Park（计算机科学系马里兰大学- College Park）；Department of Computer Science and Engineering University of Texas at Arlington（计算机科学与工程系德克萨斯理工大学阿灵顿分校）
Exploring Sparsity for Parameter Efficient Fine Tuning Using Wavelets for Vision
Dept. of Electrical and Electronics Engineering, Koç University（电子工程系，科奇大学）；Codeway AI Research（Codeway人工智能研究）；Dept. of Computer Engineering, Middle East Technical University（计算机工程系，中东技术大学）
该研究提出小波微调（WaveFT），利用权重矩阵小波域的稀疏更新实现细粒度参数控制，在视觉任务的参数高效微调方法中达最优，已纳入Hugging Face PEFT库。
Random Masking Finds Winning Tickets for Parameter Efficient Fine-tuning
Institute for Interdisciplinary Information Sciences, Tsinghua University（清华大学交叉信息研究院）；Shanghai Qizhi Institute（上海期智研究院）；Shanghai AI Laboratory（上海人工智能实验室）
