arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2212.08045cs.CV
收藏

CLIPPO:仅从像素进行图像与语言理解

CLIPPO: Image-and-Language Understanding from Pixels Only

Google Research(谷歌研究院)

Michael Tschannen, Basil Mustafa, Neil Houlsby

AI总结:

CLIPPO 是一种纯基于像素的多模态模型,仅使用单一编码器和对比损失处理图像与渲染为图像的文本,在参数量减半的情况下于多模态检索、零样本分类及视觉问答等任务上取得优异表现,且无需分词器。

摘要:

多模态模型正变得越来越有效,部分原因在于诸如 Transformer 架构等统一组件。然而,多模态模型通常仍由许多特定于任务和模态的组件及训练流程组成。例如,CLIP (Radford et al., 2021) 通过对比损失训练独立的文本和图像塔。我们探索了一种额外的统一方式:使用纯基于像素的模型来执行图像、文本和多模态任务。我们的模型仅使用对比损失进行训练,因此我们将其称为 CLIP-Pixels Only (CLIPPO)。CLIPPO 使用单一编码器处理常规图像和渲染为图像的文本。CLIPPO 在检索和零样本图像分类等基于图像的任务上表现几乎与 CLIP 风格模型一样好,且参数量仅为后者的一半,没有特定于文本的塔或嵌入。当通过图像-文本对比学习和下一句对比学习进行联合训练时,CLIPPO 能在自然语言理解任务上表现良好,且无需任何词级别的损失(语言建模或掩码语言建模),优于基于像素的先前工作。令人惊讶的是,CLIPPO 仅通过将问题和图像一起渲染,就能在视觉问答中获得良好的准确率。最后,我们利用 CLIPPO 不需要分词器这一事实,表明它无需修改即可在多语言多模态检索上取得强劲表现。

英文摘要:

Multimodal models are becoming increasingly effective, in part due to unified components, such as the Transformer architecture. However, multimodal models still often consist of many task- and modality-specific pieces and training procedures. For example, CLIP (Radford et al., 2021) trains independent text and image towers via a contrastive loss. We explore an additional unification: the use of a pure pixel-based model to perform image, text, and multimodal tasks. Our model is trained with contrastive loss alone, so we call it CLIP-Pixels Only (CLIPPO). CLIPPO uses a single encoder that processes both regular images and text rendered as images. CLIPPO performs image-based tasks such as retrieval and zero-shot image classification almost as well as CLIP-style models, with half the number of parameters and no text-specific tower or embedding. When trained jointly via image-text contrastive learning and next-sentence contrastive learning, CLIPPO can perform well on natural language understanding tasks, without any word-level loss (language modelling or masked language modelling), outperforming pixel-based prior work. Surprisingly, CLIPPO can obtain good accuracy in visual question answering, simply by rendering the question and image together. Finally, we exploit the fact that CLIPPO does not require a tokenizer to show that it can achieve strong performance on multilingual multimodal retrieval without modifications.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑