Florence-VL：利用生成式视觉编码器与深度-广度融合增强视觉-语言模型
Florence-VL: Enhancing Vision-Language Models with Generative Vision Encoder and Depth-Breadth Fusion
AI总结：
本文提出 Florence-VL，通过 Florence-2 生成式视觉编码器及 DBFusion 融合多深度、多提示视觉特征，并结合两阶段训练提升多模态大模型性能。
摘要：
我们提出了 Florence-VL，这是一个新的多模态大语言模型（MLLM）系列，其丰富的视觉表示由生成式视觉基础模型 Florence-2 产生。与通过对比学习训练、被广泛使用的 CLIP 式视觉 Transformer 不同，Florence-2 能够捕获不同层级和方面的视觉特征，这些特征更通用，可适配于多种下游任务。我们提出了一种新颖的特征融合架构和创新训练方案，可将 Florence-2 的视觉特征有效集成到 Phi 3.5 和 LLama 3 等预训练 LLM 中。具体而言，我们提出“depth-breath fusion（DBFusion，深度-广度融合）”，用于融合从不同深度提取并在多个提示下获得的视觉特征。模型训练包括先对整个模型进行端到端预训练，随后对投影层和 LLM 进行微调；训练使用精心设计的多样化开源数据集方案，其中包含高质量图像描述和指令微调对。我们对 Florence-VL 视觉特征的定量分析与可视化表明，其在视觉-语言对齐方面优于流行视觉编码器，而丰富的深度和广度在其中发挥了重要作用。在涵盖通用 VQA、感知、幻觉、OCR、图表、知识密集型理解等多种多模态和以视觉为中心的基准上，Florence-VL 相比现有最先进 MLLM 取得了显著提升。为促进未来研究，我们的模型和完整训练方案已开源：https://github.com/JiuhaiChen/Florence-VL。
英文摘要：
We present Florence-VL, a new family of multimodal large language models (MLLMs) with enriched visual representations produced by Florence-2, a generative vision foundation model. Unlike the widely used CLIP-style vision transformer trained by contrastive learning, Florence-2 can capture different levels and aspects of visual features, which are more versatile to be adapted to diverse downstream tasks. We propose a novel feature-fusion architecture and an innovative training recipe that effectively integrates Florence-2's visual features into pretrained LLMs, such as Phi 3.5 and LLama 3. In particular, we propose "depth-breath fusion (DBFusion)" to fuse the visual features extracted from different depths and under multiple prompts. Our model training is composed of end-to-end pretraining of the whole model followed by finetuning of the projection layer and the LLM, on a carefully designed recipe of diverse open-source datasets that include high-quality image captions and instruction-tuning pairs. Our quantitative analysis and visualization of Florence-VL's visual features show its advantages over popular vision encoders on vision-language alignment, where the enriched depth and breath play important roles. Florence-VL achieves significant improvements over existing state-of-the-art MLLMs across various multi-modal and vision-centric benchmarks covering general VQA, perception, hallucination, OCR, Chart, knowledge-intensive understanding, etc. To facilitate future research, our models and the complete training recipe are open-sourced. https://github.com/JiuhaiChen/Florence-VL
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
