In finetuning a large pretrained model to downstream tasks, parameter-efficient fine-tuning (PEFT) methods can effectively finetune pretrained models with few trainable parameters, but suffer from high GPU memory consumption and slow training speed. Because learnable parameters from these methods are entangled with the pretrained model, gradients related to the frozen pretrained model's parameters have to be computed and stored during finetuning. We propose Low-rank Attention Side-Tuning (LAST), which disentangles the trainable module from the pretrained model by freezing not only parameters but also outputs of the pretrained network. LAST trains a side-network composed of only low-rank self-attention modules. By viewing the pretrained model as a frozen feature extractor, the side-network takes intermediate output from the pretrained model and focus on learning task-specific knowledge. We also show that LAST can be highly parallel across multiple optimization objectives, making it very efficient in downstream task adaptation, for example, in finding optimal hyperparameters. LAST outperforms previous state-of-the-art methods on VTAB-1K and other visual adaptation tasks with roughly only 30\% of GPU memory footprint and 60\% of training time compared to existing PEFT methods, but achieves significantly higher accuracy.
An Universal Image Attractiveness Ranking Framework
Microsoft（微软公司）
我们提出了一个新的框架，使用一个新的成对深度网络来对图像吸引力进行排名，该网络使用来自Web图像索引的大量并排多标记图像对进行训练。评委只提供两幅图像之间的相对排名，而不需要直接分配绝对分数，也不需要对任何预定义的图像属性进行评级，从而使评级更加直观和准确。我们研究了深度吸引力排名网络（DARN），深度卷积神经网络和排名网络的组合，以直接学习每个图像的吸引力得分均值和方差以及法官用于标记每对的基本标准。该模型的扩展（DARN-V2）能够适应法官个人的偏好。我们还表明，在一个真正的商业搜索引擎中使用这种吸引力的信息，搜索结果的吸引力显着提高。我们评估我们的模型对其他国家的最先进的模型对我们的并排网络测试数据和另一个公共美学数据集。在判断少得多的情况下（1 M vs 50 M），我们的模型在并排标记的数据上表现出色，并且在绝对分数标记的数据上具有可比性。
M2IST: Multi-Modal Interactive Side-Tuning for Efficient Referring Expression Comprehension
College of Electronics and Information Engineering, Sichuan University（四川大学电子信息学院）；College of Systems Engineering, National University of Defense Technology（国防科技大学系统工程学院）；State Key Laboratory for Novel Software Technology, Nanjing University（南京大学计算机软件新技术国家重点实验室）；School of Engineering, Westlake University（西湖大学工程学院）；College of Computer Science and Cyber Security (Pilot Software College), Chengdu University of Technology（成都理工大学计算机与网络安全学院（软件学院））；Yunnan Key Laboratory of Software Engineering, Yunnan University（云南大学云南省软件工程重点实验室）
Beyond Low-Rank Tuning: Model Prior-Guided Rank Allocation for Effective Transfer in Low-Data and Large-Gap Regimes
School of Automation and Intelligent Sensing, Institute of Medical Robotics, Institute of Image Processing and Pattern Recognition, Shanghai Jiao Tong University（自动化与智能感知学院、医疗机器人研究院、图像处理与模式识别研究院、上海交通大学）
LoLDU: Low-Rank Adaptation via Lower-Diag-Upper Decomposition for Parameter-Efficient Fine-Tuning
University of Electronic Science and Technology of China（电子科技大学）；Center for Future Media（未来媒体中心）；School of Computer Science and Engineering（计算机科学与工程学院）
