MiniVLM：一种更小更快的视觉语言模型
MiniVLM: A Smaller and Faster Vision-Language Model
摘要：
最近的视觉语言（VL）的研究已经取得了显着的进展，学习通用表示与Transformer模型，然后微调下游VL任务的大量图像-文本对。虽然现有的研究一直专注于通过大型预训练模型实现高准确性，但构建轻量级模型在实践中具有很大的价值，但探索较少。在本文中，我们提出了一个更小，更快的VL模型，MiniVLM，它可以微调，在各种下游任务的良好性能，就像它的大型同行。MiniVLM由两个模块组成，一个视觉特征提取器和一个基于变换的视觉语言融合模块。我们设计了一个两阶段有效的特征提取器（TEE），灵感来自一个阶段的EfficientDet网络，显着减少视觉特征提取的时间成本$95\%$，相比基线模型。通过比较不同的紧凑型BERT模型，采用MiniLM结构来降低Transformer模块的计算量。此外，我们通过添加$7M$ Open Images数据来改进MiniVLM预训练，这些数据由最先进的字幕模型进行伪标记。我们还使用从强标记模型中获得的高质量图像标记进行预训练，以增强跨模态对齐。大型模型可以离线使用，而不会在微调和推理中增加任何开销。通过上述设计选择，我们的MiniVLM通过$73\%$减少了模型大小，通过$94\%$减少了推理时间成本，同时能够在多个VL任务上保持$94-97\%$的准确性。我们希望MiniVLM有助于简化最先进的VL研究在边缘应用中的使用。
英文摘要：
Recent vision-language (VL) studies have shown remarkable progress by learning generic representations from massive image-text pairs with transformer models and then fine-tuning on downstream VL tasks. While existing research has been focused on achieving high accuracy with large pre-trained models, building a lightweight model is of great value in practice but is less explored. In this paper, we propose a smaller and faster VL model, MiniVLM, which can be finetuned with good performance on various downstream tasks like its larger counterpart. MiniVLM consists of two modules, a vision feature extractor and a transformer-based vision-language fusion module. We design a Two-stage Efficient feature Extractor (TEE), inspired by the one-stage EfficientDet network, to significantly reduce the time cost of visual feature extraction by $95\%$, compared to a baseline model. We adopt the MiniLM structure to reduce the computation cost of the transformer module after comparing different compact BERT models. In addition, we improve the MiniVLM pre-training by adding $7M$ Open Images data, which are pseudo-labeled by a state-of-the-art captioning model. We also pre-train with high-quality image tags obtained from a strong tagging model to enhance cross-modality alignment. The large models are used offline without adding any overhead in fine-tuning and inference. With the above design choices, our MiniVLM reduces the model size by $73\%$ and the inference time cost by $94\%$ while being able to retain $94-97\%$ of the accuracy on multiple VL tasks. We hope that MiniVLM helps ease the use of the state-of-the-art VL research for on-the-edge applications.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
