Ferret-v2: 一种面向大语言模型指代与定位任务的改进基线
Ferret-v2: An Improved Baseline for Referring and Grounding with Large Language Models
AI总结：
针对原有Ferret模型受固定视觉编码器限制、泛化性不足的问题，研究推出升级模型Ferret-v2，通过任意分辨率处理、多粒度DINOv2编码、三阶段训练三项设计，在指代与定位任务上显著超越原有模型及其他前沿方法。
摘要：
尽管Ferret将区域理解无缝集成到大语言模型（LLM）中以提升其指代与定位能力，但它仍存在一定局限性：受限于预训练的固定视觉编码器，且无法在更广泛的任务上取得良好表现。在本研究中，我们推出Ferret-v2——Ferret的重大升级版本，包含三项核心设计。（1）任意分辨率定位与指代：一种可灵活处理更高图像分辨率的方法，提升了模型更细致地处理和理解图像的能力。（2）多粒度视觉编码：通过集成额外的DINOv2编码器，模型能为全局和细粒度视觉信息学习更优质、多样的底层上下文。（3）三阶段训练范式：除图像-文本对齐外，在最终指令微调前新增一个高分辨率密集对齐阶段。实验表明，得益于高分辨率扩展与细粒度视觉处理，Ferret-v2相较Ferret及其他前沿方法实现了大幅性能提升。
英文摘要：
While Ferret seamlessly integrates regional understanding into the Large Language Model (LLM) to facilitate its referring and grounding capability, it poses certain limitations: constrained by the pre-trained fixed visual encoder and failed to perform well on broader tasks. In this work, we unveil Ferret-v2, a significant upgrade to Ferret, with three key designs. (1) Any resolution grounding and referring: A flexible approach that effortlessly handles higher image resolution, improving the model's ability to process and understand images in greater detail. (2) Multi-granularity visual encoding: By integrating the additional DINOv2 encoder, the model learns better and diverse underlying contexts for global and fine-grained visual information. (3) A three-stage training paradigm: Besides image-caption alignment, an additional stage is proposed for high-resolution dense alignment before the final instruction tuning. Experiments show that Ferret-v2 provides substantial improvements over Ferret and other state-of-the-art methods, thanks to its high-resolution scaling and fine-grained visual processing.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
