arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2310.09199cs.CV
收藏

PaLI-3 视觉语言模型:更小、更快、更强

PaLI-3 Vision Language Models: Smaller, Faster, Stronger

Google Research(谷歌研究院);Google DeepMind(谷歌DeepMind);Google Cloud(谷歌云)
Xi Chen, Xiao Wang, Lucas Beyer, Alexander Kolesnikov, Jialin Wu, Paul Voigtlaender, Basil Mustafa, Sebastian Goodman, Ibrahim Alabdulmohsin, Piotr Padlewski, D… 展开作者

Xi Chen, Xiao Wang, Lucas Beyer, Alexander Kolesnikov, Jialin Wu, Paul Voigtlaender, Basil Mustafa, Sebastian Goodman, Ibrahim Alabdulmohsin, Piotr Padlewski, Daniel Salz, Xi Xiong, Daniel Vlasic, Filip Pavetic, Keran Rong, Tianli Yu, Daniel Keysers, Xiaohua Zhai, Radu Soricut

AI总结:

本文提出 PaLI-3 视觉语言模型,通过采用对比预训练的 SigLIP 图像编码器,在定位和多模态任务上表现优异,以仅 5B 参数超越了 10 倍大的模型并达到 SOTA。

摘要:

本文提出了 PaLI-3,一个更小、更快且更强的视觉语言模型(VLM),其表现优于大 10 倍的同类模型。为了实现这一强劲性能,我们将使用分类目标预训练的 Vision Transformer(ViT)模型与对比(SigLIP)预训练的模型进行了比较。我们发现,尽管在标准图像分类基准上表现稍逊,但基于 SigLIP 的 PaLI 在各种多模态基准上表现出色,尤其是在定位和视觉情境文本理解方面。我们将 SigLIP 图像编码器扩展到 20 亿参数,并在多语言跨模态检索上取得了新的 SOTA。我们希望仅有 5B 参数的 PaLI-3 能重新激发对复杂 VLM 基础组件的研究,并推动新一代扩展模型的发展。

英文摘要:

This paper presents PaLI-3, a smaller, faster, and stronger vision language model (VLM) that compares favorably to similar models that are 10x larger. As part of arriving at this strong performance, we compare Vision Transformer (ViT) models pretrained using classification objectives to contrastively (SigLIP) pretrained ones. We find that, while slightly underperforming on standard image classification benchmarks, SigLIP-based PaLI shows superior performance across various multimodal benchmarks, especially on localization and visually-situated text understanding. We scale the SigLIP image encoder up to 2 billion parameters, and achieves a new state-of-the-art on multilingual cross-modal retrieval. We hope that PaLI-3, at only 5B parameters, rekindles research on fundamental pieces of complex VLMs, and could fuel a new generation of scaled-up models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑