DAVE: 一种用于文档理解与网络代理的视觉编码器
DAVE: A VLM Vision Encoder for Document Understanding and Web Agents
AI总结：
DAVE是一种专为文档理解和网络代理设计的视觉编码器，通过自监督和监督预训练结合模型融合策略，提升对文档和网络任务的适应性与性能。
摘要：
尽管视觉-语言模型（VLMs）在多模态任务中表现出色，但其选择的视觉编码器存在根本性缺陷：低层特征缺乏文档理解和网络代理所需的稳健结构和空间信息。为弥合这一差距，我们引入了DAVE，一种为VLMs量身定制、针对这些任务优化的视觉编码器。我们的训练流程旨在利用大量未标记数据，以避免对文档和网络图像进行昂贵的大规模标注的需要。我们从在未标记图像上进行自监督预训练开始，随后进行监督性自回归预训练阶段，其中模型从有限的高质量数据中学习解析和定位等任务。在监督阶段，我们采用两种策略来提高编码器与通用视觉知识以及多样化的文档和网络代理任务的对齐：（i）我们引入了一种新的模型融合方案，结合使用不同文本解码器训练的编码器，以确保与不同网络代理架构的广泛兼容性。（ii）我们使用集成训练将预训练的通用编码器（如SigLIP2）的特征与我们自己的文档和网络特定表示融合。在经典文档任务、VQAs、网络定位和基于代理的基准测试中进行了广泛的实验，验证了我们方法的有效性，确立了DAVE作为文档和网络应用的强大视觉编码器。
英文摘要：
While Vision-language models (VLMs) have demonstrated remarkable performance across multi-modal tasks, their choice of vision encoders presents a fundamental weakness: their low-level features lack the robust structural and spatial information essential for document understanding and web agents. To bridge this gap, we introduce DAVE, a vision encoder purpose-built for VLMs and tailored for these tasks. Our training pipeline is designed to leverage abundant unlabeled data to bypass the need for costly large-scale annotations for document and web images. We begin with a self-supervised pretraining stage on unlabeled images, followed by a supervised autoregressive pretraining stage, where the model learns tasks like parsing and localization from limited, high-quality data. Within the supervised stage, we adopt two strategies to improve our encoder's alignment with both general visual knowledge and diverse document and web agentic tasks: (i) We introduce a novel model-merging scheme, combining encoders trained with different text decoders to ensure broad compatibility with different web agentic architectures. (ii) We use ensemble training to fuse features from pretrained generalist encoders (e.g., SigLIP2) with our own document and web-specific representations. Extensive experiments on classic document tasks, VQAs, web localization, and agent-based benchmarks validate the effectiveness of our approach, establishing DAVE as a strong vision encoder for document and web applications.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
