arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.02055cs.LGcs.AI
收藏

方差对齐分数:多模态对比学习中一种简单但难以超越的数据选择方法

Variance Alignment Score: A Simple But Tough-to-Beat Data Selection Method for Multimodal Contrastive Learning

University of Washington(华盛顿大学);Paul G. Allen School of Computer Science & Engineering(保罗·G·艾伦计算机科学与工程学院)

Yiping Wang, Yifang Chen, Wendan Yan, Kevin Jamieson, Simon Shaolei Du

AI总结:

提出方差对齐分数(VAS)作为多模态对比学习的数据选择指标,通过最大化与目标协方差矩阵的对齐来选择信息量最大的样本,结合CLIP分数在DataComp和CC12M上分别提升1.3%和2.5%的平均性能。

摘要:

近年来,数据选择已成为大规模视觉-语言模型预训练的核心问题,尤其是在处理来自网络的噪声数据集时。一种广泛采用的策略是为每个样本分配质量分数(如CLIP相似度),并保留得分最高的数据对。然而,这些方法不考虑数据分布,且总是无法选择信息量最大的样本。为解决此问题,我们提出了一种简单但具有理论依据的度量指标,名为方差对齐分数(VAS),其形式为$\langle \Sigma_{\text{test}}, \Sigma_i\rangle$。其中,$\Sigma_{\text{test}}$表示我们旨在对齐的目标(交叉)协方差矩阵,可能基于先验知识,而$\Sigma_i$表示第$i$个样本的单模态或多模态表示的张量积。我们进一步设计了一种新的数据选择方法,最大化总VAS。我们在简化设置中提供了理论分析,以证明VAS相对于随机选择或其他现有数据选择方法的理论优势。实验上,将VAS和CLIP分数结合使用,在噪声数据集DataComp的38个评估集上平均超过基线1.3%,在高质量数据集CC12M的VTAB上超过基线2.5%。此外,我们的消融研究还表明,在计算VAS时视觉特征优于文本特征,且相关的经典实验设计方法在此背景下可能失效。

英文摘要:

In recent years, data selection has emerged as a core issue for large-scale visual-language model pretraining, especially on noisy web-curated datasets. One widely adopted strategy assigns quality scores such as CLIP similarity for each sample and retains the data pairs with the highest scores. However, these approaches are agnostic of data distribution and always fail to select the most informative samples. To solve this problem, we propose a simple yet theoretically principled metric named Variance Alignment Score (VAS), which has the form $\langle Σ_{\text{test}}, Σ_i\rangle$. Here, $Σ_{\text{test}}$ represents the target (cross-)covariance matrix we aim to align, potentially based on prior knowledge, while $Σ_i$ denotes the tensor product of single or multi-modal representations for the $i$-th sample. We further design a new data selection method that maximizes the total VAS. We provide theoretical analysis in a simplified setting to demonstrate the theoretical advantage of VAS over random or other existing data selection. Experimentally, applying VAS and CLIP scores together can outperform baselines by a margin of $1.3\%$ average on 38 evaluation sets for noisy dataset DataComp and $2.5\%$ on VTAB for high-quality dataset CC12M. Additionally, our ablation study also shows visual features are better than text for calculating VAS, and the related classical experimental design methods may fail under this context.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑