arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2408.01099cs.CVcs.AI
收藏

基于贡献度的低秩适配与预训练模型用于真实图像复原

Contribution-based Low-Rank Adaptation with Pre-training Model for Real Image Restoration

IPAI;INMC;Seoul National University(首尔大学)

Donwon Park, Hayeon Kim, Se Young Chun

AI总结:

本文提出CoLoRA方法,通过基于贡献度的低秩适配仅微调少量参数,并配合随机顺序退化预训练策略PROD,在多种真实图像复原任务上以高效参数利用取得与全量微调相当甚至更优的性能。

摘要:

近年来,预训练模型和高效参数微调在自然语言处理和高层计算机视觉中借助掩码建模与提示微调取得了显著成功。然而,在低层计算机视觉中,关于预训练模型的研究仍然有限,甚至高效微调策略也尚未被探索,尽管它在各种实际任务中具有重要性和益处,例如在AI边缘设备上集成新任务时缓解内存膨胀问题。在此,我们提出了一种新颖的高效参数微调方法,称为基于贡献度的低秩适配(CoLoRA),用于多种图像复原任务,并配合一种有效的随机顺序退化预训练方法(PROD)。与先前微调所有网络参数的方法不同,我们的CoLoRA通过利用LoRA(低秩适配)为每个新视觉任务有效微调少量参数,并采用基于贡献度的方法逐层自适应地确定该任务的容量,从而获得与全量微调相当的性能。此外,我们的PROD策略能够扩展预训练模型的能力,提升性能与鲁棒性,以弥合合成预训练与真实世界微调之间的差距。我们的CoLoRA结合PROD在多种图像复原任务中展现出优越性能,覆盖合成与真实世界数据集上的多种退化类型,适用于已知任务和新任务。

英文摘要:

Recently, pre-trained model and efficient parameter tuning have achieved remarkable success in natural language processing and high-level computer vision with the aid of masked modeling and prompt tuning. In low-level computer vision, however, there have been limited investigations on pre-trained models and even efficient fine-tuning strategy has not yet been explored despite its importance and benefit in various real-world tasks such as alleviating memory inflation issue when integrating new tasks on AI edge devices. Here, we propose a novel efficient parameter tuning approach dubbed contribution-based low-rank adaptation (CoLoRA) for multiple image restorations along with effective pre-training method with random order degradations (PROD). Unlike prior arts that tune all network parameters, our CoLoRA effectively fine-tunes small amount of parameters by leveraging LoRA (low-rank adaptation) for each new vision task with our contribution-based method to adaptively determine layer by layer capacity for that task to yield comparable performance to full tuning. Furthermore, our PROD strategy allows to extend the capability of pre-trained models with improved performance as well as robustness to bridge synthetic pre-training and real-world fine-tuning. Our CoLoRA with PROD has demonstrated its superior performance in various image restoration tasks across diverse degradation types on both synthetic and real-world datasets for known and novel tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑