PaliGemma 2：一系列用于迁移的多功能 VLM
PaliGemma 2: A Family of Versatile VLMs for Transfer
AI总结：
本文提出 PaliGemma 2，将 SigLIP-So400m 与 2B 至 27B 的 Gemma 2 结合，并在三种分辨率下分阶段训练，系统分析迁移因素且在多类视觉任务上达到 SOTA。
摘要：
PaliGemma 2 是对基于 Gemma 2 语言模型系列的开放视觉-语言模型（VLM）PaliGemma 的升级。我们将 PaliGemma 同样使用过的 SigLIP-So400m 视觉编码器与全系列 Gemma 2 模型相结合，从 2B 模型一直到 27B 模型。我们以三种分辨率（224px、448px 和 896px）分多个阶段训练这些模型，使其具备通过微调进行迁移所需的广泛知识。由此得到的涵盖不同模型规模和分辨率的基础模型系列，使我们能够研究影响迁移性能的因素（例如学习率），并分析任务类型、模型规模与分辨率之间的相互作用。我们还进一步增加了 PaliGemma 范围之外的迁移任务数量和广度，包括表格结构识别、分子结构识别、乐谱识别等不同的 OCR 相关任务，以及长细粒度描述生成和放射影像报告生成；在这些任务上，PaliGemma 2 取得了当前最优结果。
英文摘要：
PaliGemma 2 is an upgrade of the PaliGemma open Vision-Language Model (VLM) based on the Gemma 2 family of language models. We combine the SigLIP-So400m vision encoder that was also used by PaliGemma with the whole range of Gemma 2 models, from the 2B one all the way up to the 27B model. We train these models at three resolutions (224px, 448px, and 896px) in multiple stages to equip them with broad knowledge for transfer via fine-tuning. The resulting family of base models covering different model sizes and resolutions allows us to investigate factors impacting transfer performance (such as learning rate) and to analyze the interplay between the type of task, model size, and resolution. We further increase the number and breadth of transfer tasks beyond the scope of PaliGemma including different OCR-related tasks such as table structure recognition, molecular structure recognition, music score recognition, as well as long fine-grained captioning and radiography report generation, on which PaliGemma 2 obtains state-of-the-art results.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
