arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2303.11403cs.CVcs.CLcs.LG
收藏

eP-ALM:语言模型的高效感知增强

eP-ALM: Efficient Perceptual Augmentation of Language Models

Sorbonne University(索邦大学)

Mustafa Shukor, Corentin Dancette, Matthieu Cord

AI总结:

本文提出eP-ALM方法,通过冻结99%以上参数、仅训练一个线性投影层并前置一个可训练token,以极低计算成本将单模态预训练模型高效适配到图像、视频和音频的多模态VQA与Captioning任务,并显著优于现有基线。

摘要:

大型语言模型(LLMs)迄今为止给世界留下了深刻印象,在大规模模型中展现出前所未有的能力。在视觉方面,Transformer模型(即ViT)正遵循同样的趋势,在具有挑战性的基准上取得了最佳性能。随着此类单模态模型的大量涌现,一个自然的问题随之而来:我们是否也需要遵循这一趋势来处理多模态任务?在这项工作中,我们建议将努力转向对现有模型的高效适配,并提出用感知能力来增强语言模型。现有的针对视觉-语言任务适配预训练模型的方法仍然依赖几个阻碍其效率的关键组件。特别是,它们仍然训练大量参数,依赖大规模多模态预训练,使用在庞大图像-文本数据集上训练的编码器(例如CLIP),并增加显著的推理开销。此外,这些方法大多专注于零样本(Zero-Shot)和上下文学习(In Context Learning),很少甚至没有在直接微调方面做出努力。我们研究了将单模态模型适配到多模态任务所需的最小计算工作量,并提出了一个新的具有挑战性的设置,以及不同的方法,以高效地适配单模态预训练模型。我们表明,通过冻结超过99%的总参数、仅训练一个线性投影层,并且仅前置一个可训练token,我们的方法(称为eP-ALM)在所提出的设置下,在图像、视频和音频模态上的VQA和Captioning任务中显著优于其他基线。代码可在以下地址获取:https://github.com/mshukor/eP-ALM。

英文摘要:

Large Language Models (LLMs) have so far impressed the world, with unprecedented capabilities that emerge in models at large scales. On the vision side, transformer models (i.e., ViT) are following the same trend, achieving the best performance on challenging benchmarks. With the abundance of such unimodal models, a natural question arises; do we need also to follow this trend to tackle multimodal tasks? In this work, we propose to rather direct effort to efficient adaptations of existing models, and propose to augment Language Models with perception. Existing approaches for adapting pretrained models for vision-language tasks still rely on several key components that hinder their efficiency. In particular, they still train a large number of parameters, rely on large multimodal pretraining, use encoders (e.g., CLIP) trained on huge image-text datasets, and add significant inference overhead. In addition, most of these approaches have focused on Zero-Shot and In Context Learning, with little to no effort on direct finetuning. We investigate the minimal computational effort needed to adapt unimodal models for multimodal tasks and propose a new challenging setup, alongside different approaches, that efficiently adapts unimodal pretrained models. We show that by freezing more than 99% of total parameters, training only one linear projection layer, and prepending only one trainable token, our approach (dubbed eP-ALM) significantly outperforms other baselines on VQA and Captioning across Image, Video, and Audio modalities, following the proposed setup. The code is available here: https://github.com/mshukor/eP-ALM.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑