统一词汇表示用于可解释的视觉-语言对齐
Unified Lexical Representation for Interpretable Visual-Language Alignment
AI总结:
本文提出LexVLA,通过统一词汇表示实现可解释的视觉-语言对齐,利用DINOv2和Llama 2,无需复杂设计,在跨模态检索上优于更大规模数据训练的基线。
摘要:
视觉-语言对齐(VLA)自CLIP的开创性工作以来受到了广泛关注。尽管CLIP表现良好,但其典型的直接潜在特征对齐在表示和相似度得分方面缺乏清晰度。另一方面,词汇表示是一种向量,其元素表示样本与词汇表中某个单词之间的相似度,是一种自然的稀疏表示且具有可解释性,能够为单个单词提供精确匹配。然而,由于缺乏真实标签监督和虚假发现问题,词汇表示难以学习,因此需要复杂的设计才能有效训练。在本文中,我们引入了LexVLA,一个更具可解释性的VLA框架,通过为两种模态学习统一的词汇表示,无需复杂设计。我们使用DINOv2作为视觉模型,因其具有局部倾向的特征,并使用Llama 2,一个生成式语言模型,以利用其上下文词汇预测能力。为了避免虚假发现,我们提出了一种过度使用惩罚,以抑制词汇表示错误地频繁激活无意义的单词。我们证明,这两个预训练的单模态模型可以通过在适度的多模态数据集上进行微调而良好对齐,并避免复杂的训练配置。在跨模态检索基准上,LexVLA在CC-12M多模态数据集上训练,优于在更大数据集(如YFCC15M)上微调的基线,以及从头开始在更大数据集(如1.1B数据,包括CC-12M)上训练的基线。我们进行了大量实验来分析LexVLA。代码可在https://github.com/Clementine24/LexVLA获取。
英文摘要:
Visual-Language Alignment (VLA) has gained a lot of attention since CLIP's groundbreaking work. Although CLIP performs well, the typical direct latent feature alignment lacks clarity in its representation and similarity scores. On the other hand, lexical representation, a vector whose element represents the similarity between the sample and a word from the vocabulary, is a natural sparse representation and interpretable, providing exact matches for individual words. However, lexical representations are difficult to learn due to no ground-truth supervision and false-discovery issues, and thus requires complex design to train effectively. In this paper, we introduce LexVLA, a more interpretable VLA framework by learning a unified lexical representation for both modalities without complex design. We use DINOv2 as our visual model for its local-inclined features and Llama 2, a generative language model, to leverage its in-context lexical prediction ability. To avoid the false discovery, we propose an overuse penalty to refrain the lexical representation from falsely frequently activating meaningless words. We demonstrate that these two pre-trained uni-modal models can be well-aligned by fine-tuning on the modest multi-modal dataset and avoid intricate training configurations. On cross-modal retrieval benchmarks, LexVLA, trained on the CC-12M multi-modal dataset, outperforms baselines fine-tuned on larger datasets (e.g., YFCC15M) and those trained from scratch on even bigger datasets (e.g., 1.1B data, including CC-12M). We conduct extensive experiments to analyze LexVLA. Codes are available at https://github.com/Clementine24/LexVLA.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。