BLIP-Adapter：面向移动屏幕截图描述生成的参数高效迁移学习
BLIP-Adapter: Parameter-Efficient Transfer Learning for Mobile Screenshot Captioning
AI总结：
针对移动屏幕截图描述生成中全量微调成本高的问题，本文提出BLIP-Adapter，通过冻结主干并训练组合适配器实现参数高效迁移，性能可比全量微调且显著减少参数量。
摘要：
本研究旨在探索用于屏幕截图描述生成任务的高效微调方法。近年来，图像描述生成取得了显著进展，但针对移动屏幕的描述生成任务研究仍然相对匮乏。目前用于描述产品屏幕截图中用户行为的数据集和用例明显有限。因此，我们试图针对屏幕截图描述生成任务微调已有的模型。然而，由于图像描述生成模型包含大量参数，微调大型预训练模型可能会耗费大量资源，需要相当多的时间、计算能力和存储空间。为应对这一挑战，本研究提出了一种适配器方法的组合，该方法只需微调模型上的附加模块。这些方法最初是为视觉或语言任务设计的，我们的意图是将其应用于解决屏幕截图描述生成中的类似挑战。通过冻结图像描述模型的参数，仅训练与这些方法相关的权重，可以实现与微调整个模型相当的性能，同时显著减少参数数量。本研究首次全面考察了在屏幕截图描述生成任务背景下组合适配器的有效性。通过实验和分析，本研究旨在为适配器在视觉-语言模型中的应用提供有价值的见解，并推动屏幕截图描述生成任务的高效微调技术发展。我们的研究可在 https://github.com/RainYuGG/BLIP-Adapter 获取。
英文摘要：
This study aims to explore efficient tuning methods for the screenshot captioning task. Recently, image captioning has seen significant advancements, but research in captioning tasks for mobile screens remains relatively scarce. Current datasets and use cases describing user behaviors within product screenshots are notably limited. Consequently, we sought to fine-tune pre-existing models for the screenshot captioning task. However, fine-tuning large pre-trained models can be resource-intensive, requiring considerable time, computational power, and storage due to the vast number of parameters in image captioning models. To tackle this challenge, this study proposes a combination of adapter methods, which necessitates tuning only the additional modules on the model. These methods are originally designed for vision or language tasks, and our intention is to apply them to address similar challenges in screenshot captioning. By freezing the parameters of the image caption models and training only the weights associated with the methods, performance comparable to fine-tuning the entire model can be achieved, while significantly reducing the number of parameters. This study represents the first comprehensive investigation into the effectiveness of combining adapters within the context of the screenshot captioning task. Through our experiments and analyses, this study aims to provide valuable insights into the application of adapters in vision-language models and contribute to the development of efficient tuning techniques for the screenshot captioning task. Our study is available at https://github.com/RainYuGG/BLIP-Adapter
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
