arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2208.02532cs.CL
收藏

生成式多模态预训练模型的提示微调

Prompt Tuning for Generative Multimodal Pretrained Models

DAMO Academy, Alibaba Group(阿里巴巴集团达摩院)

Hao Yang, Junyang Lin, An Yang, Peng Wang, Chang Zhou, Hongxia Yang

摘要:

提示微调已经成为模型调整的一种新范式,并且在自然语言预训练甚至视觉预训练中已经证明了成功。在这项工作中,我们探索了提示微调到多模态预训练的转移,重点是生成多模态预训练模型,而不是对比模型。具体来说,我们实现提示微调的统一序列到序列的预训练模型适应的理解和生成任务。实验结果表明,轻量级的提示微调可以达到与微调相当的性能,优于其他轻量级的调整方法。此外,与微调模型相比,微调模型对对抗攻击表现出更好的鲁棒性。我们进一步指出,实验因素,包括提示长度,提示深度,和重新参数化,有很大的影响模型的性能,从而我们经验性地提供了一个建议的设置提示微调。尽管提示微调具有明显的优势,但仍存在一些局限性,并指出了今后的研究方向。代码可从\url{https://github.com/OFA-Sys/OFA}获得

英文摘要:

Prompt tuning has become a new paradigm for model tuning and it has demonstrated success in natural language pretraining and even vision pretraining. In this work, we explore the transfer of prompt tuning to multimodal pretraining, with a focus on generative multimodal pretrained models, instead of contrastive ones. Specifically, we implement prompt tuning on the unified sequence-to-sequence pretrained model adaptive to both understanding and generation tasks. Experimental results demonstrate that the light-weight prompt tuning can achieve comparable performance with finetuning and surpass other light-weight tuning methods. Besides, in comparison with finetuned models, the prompt-tuned models demonstrate improved robustness against adversarial attacks. We further figure out that experimental factors, including the prompt length, prompt depth, and reparameteratization, have great impacts on the model performance, and thus we empirically provide a recommendation for the setups of prompt tuning. Despite the observed advantages, we still find some limitations in prompt tuning, and we correspondingly point out the directions for future studies. Codes are available at \url{https://github.com/OFA-Sys/OFA}

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑