用于多模态对比学习的先验拟合与冻结评分器
Prior-Fitted and Frozen Scorers for Multimodal Contrastive Learning
AI总结：
该研究针对多模态对比学习，提出先验拟合评分器（PFS），发现冻结的Transformer评分器性能优于手工基线，先验拟合可进一步提升效果，表明Transformer结合因果预置可替代人工相似度函数。
摘要：
多模态对比学习通常依赖人工指定的评分函数（如余弦相似度）来量化不同模态表征间的相似度。近期已有若干此类函数被提出，用于将对比学习扩展至多模态场景。但原则上，任何能为匹配样本分配高兼容性、为不匹配样本分配低兼容性的函数都可作为评分器。受此更宽泛视角启发，我们以Transformer实例化评分器，发现即便随机初始化的模型在保持冻结状态时，也能优于强大的手工设计多模态对比基线；而与编码器联合优化评分器的效果则较差。基于该观察，我们提出先验拟合评分器（Prior-Fitted Scorer, PFS）：一种在从结构因果模型（Structural Causal Model, SCM）先验生成的合成多模态数据上预训练的Transformer评分器，之后将其迁移至真实世界数据。SCM使评分器接触到多样的依赖结构，包括共享、成对及私有信息，为高阶跨模态相似度提供结构化先验。在四个真实世界数据集上，冻结的Transformer评分器可与对比基线竞争，而先验拟合进一步提升了随机初始化的效果。更广泛而言，我们的结果表明多模态对比学习无需依赖人工相似度函数：Transformer可提供强大的归纳偏置，而因果预训练可进一步塑造这些偏置。
英文摘要：
Multimodal contrastive learning typically relies on manually specified scoring functions, such as cosine similarity, to quantify the similarity between representations from different modalities. Several such functions have been recently proposed to extend contrastive learning beyond two modalities. In principle, however, any function that assigns high compatibility to matched samples and low compatibility to mismatched ones could serve as a scorer. Motivated by this broader view, we instantiate the scorer with a Transformer and find that even a randomly initialized model, when kept frozen, can outperform strong handcrafted multimodal contrastive baselines, whereas optimizing the scorer jointly with the encoders is less effective. Building on this observation, we introduce the Prior-Fitted Scorer (PFS): a Transformer scorer pre-trained on synthetic multimodal data generated from structural causal model (SCM) priors before transferring it to real-world data. The SCM exposes the scorer to diverse dependency structures, including shared, pairwise, and private information, providing a structured prior over higher-order cross-modal similarity. Across four real-world datasets, frozen Transformer scorers are competitive to contrastive baselines, and prior-fitting further improves the random initialization. More broadly, our results suggest that multimodal contrastive learning need not rely on manual similarity functions: Transformers can provide strong inductive biases, while causal pretraining can further shape these biases.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
