语义锚定的QFormer用于高效视觉语言理解
Semantically Grounded QFormer for Efficient Vision Language Understanding
AI总结：
提出一种更高效的QFormer视觉语言对齐框架，利用QFormer潜变量直接调节冻结LLM的潜空间进行图像到文本生成，以减少预训练计算开销。
摘要：
通用视觉语言模型（VLM）近年来受到了极大的关注，这得益于它们学习丰富视觉-语言关联的能力以及广泛的零样本能力。一条非常流行的研究路线利用冻结的单模态模型，通过一个名为QFormer的可训练模块将视觉表示桥接到语言。然而，这种方法严重依赖于大规模多模态预训练，计算开销巨大。为此，我们提出了一种更高效的基于QFormer的视觉-语言对齐框架。我们的关键思想基于一个观察：QFormer的潜变量与冻结的LLM的中间潜空间对应更强。因此，我们不将QFormer的潜变量作为LLM的输入，而是改变框架，直接使用这些潜变量来调节LLM的潜空间，以进行图像到文本的生成。我们证明了我们的方法在提高视觉语言预训练效率方面相对于现有基线的有效性。
英文摘要：
General purpose Vision Language Models (VLMs) have received tremendous interest in recent years, owing to their ability to learn rich vision-language correlations as well as their broad zero-shot competencies. One immensely popular line of work utilizes frozen unimodal models, by bridging vision representations to language using a trainable module called the QFormer. However, this method relies heavily on large-scale multimodal pretraining with huge computational overheads. To that end, we propose a more efficient framework for QFormer-based vision-language alignment. Our key idea relies on the observation that QFormer latents correspond more strongly to the frozen LLM's intermediate latent space. Consequently, instead of using QFormer latents as inputs to the LLM, we alter the framework by using the latents to directly condition the LLM latent space for image-to-text generation. We demonstrate the effectiveness of our approach against existing baselines in improving the efficiency of vision-language pretraining.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
