通过多模态持续预训练增强视觉基础模型
Enhancing Vision Foundation Models via Multimodal Continual Pre-Training
AI总结：
研究通过多模态持续预训练增强视觉基础模型，提出M-CPT框架，用持续位置嵌入处理不同分辨率视觉输入，通过特征对齐目标提升视觉与文本表示一致性，实验证明该框架能提升多模态理解性能并保持标准视觉任务表现。
摘要：
视觉基础模型（VFMs）为广泛应用提供强大视觉表示。本文通过多模态训练增强现有VFMs，使其能有效处理不同分辨率视觉输入，并产生与语言表示更匹配的视觉表示。为此引入M-CPT框架，含持续位置嵌入（CPE）处理灵活视觉分辨率，以及特征对齐目标提升多模态训练中视觉与文本表示一致性。在DINOv2等模型上实验表明，M-CPT提升多模态理解性能且保持标准视觉基准测试良好表现。
英文摘要：
Vision Foundation Models (VFMs) provide strong visual representations for a wide range of applications. In this work, we enhance prevailing VFMs through multimodal training, allowing them to effectively process visual inputs at varying resolutions while producing visual representations that are better aligned with language representations, regardless of their original pre-training objectives. To this end, we introduce M-CPT, a Multimodal Continual Pre-Training framework designed to improve the understanding capability of pre-trained VFMs while preserving their strong visual representation quality. M-CPT introduces a Continual Position Embedding (CPE) for handling flexible visual resolutions, along with a feature alignment objective that improves the consistency between visual and textual representations during multimodal training. Extensive experiments on leading VFMs, including DINOv2, SigLIP, and AIMv2, demonstrate that M-CPT consistently improves multimodal understanding performance while preserving strong performance on standard vision benchmarks such as classification and segmentation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
