基于大语言模型的视觉编码器分层预训练
Hierarchical Pre-Training of Vision Encoders with Large Language Model
AI总结：
本文提出HIVE框架，通过引入视觉编码器与大语言模型间的分层交叉注意力机制，提升视觉语言对齐，改进特征融合与表征学习，实验表明其在图像分类和多模态任务中表现优异。
摘要：
计算机视觉领域通过可扩展的视觉编码器和多模态预训练框架取得了显著进展。然而，现有方法往往将视觉编码器和大语言模型（LLMs）视为独立模块，限制了层次化视觉特征的整合。本文提出HIVE（视觉编码器分层预训练），通过引入视觉编码器与LLM之间的层次化交叉注意力机制，提升视觉语言对齐。与传统方法不同，HIVE在多个层次上实现结构化特征融合，改进梯度流动和表征学习。为优化这种交互，我们引入了三阶段训练策略，逐步对齐视觉编码器与LLM，确保稳定优化和有效多模态融合。实验证明，HIVE在图像分类和各种视觉语言任务中均取得优越性能，优于基于自注意力的方法，在MME、GQA、OK-VQA和ScienceQA等基准测试中表现更佳。我们的结果突显了层次化特征整合的优势，为更高效和表达能力强的视觉语言模型铺平了道路。
英文摘要：
The field of computer vision has experienced significant advancements through scalable vision encoders and multimodal pre-training frameworks. However, existing approaches often treat vision encoders and large language models (LLMs) as independent modules, limiting the integration of hierarchical visual features. In this work, we propose HIVE (Hierarchical Pre-Training of Vision Encoders), a novel framework that enhances vision-language alignment by introducing hierarchical cross-attention between the vision encoder and LLM. Unlike conventional methods that flatten image embeddings, HIVE enables structured feature fusion across multiple layers, improving gradient flow and representation learning. To optimize this interaction, we introduce a three-stage training strategy that progressively aligns the vision encoder with the LLM, ensuring stable optimization and effective multimodal fusion. Empirical evaluations demonstrate that HIVE achieves superior performance not only in image classification but also on various vision-language tasks, outperforming self-attention-based methods in benchmarks such as MME, GQA, OK-VQA, and ScienceQA. Our results highlight the benefits of hierarchical feature integration, paving the way for more efficient and expressive vision-language models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
