非对比视觉语言学习与预测嵌入对齐
Non-Contrastive Vision-Language Learning with Predictive Embedding Alignment
AI总结：
NOVA通过联合嵌入预测与分布正则化，实现非对比视觉语言学习，简化训练目标并提升稳定性与效果。
摘要：
视觉语言模型已改变了多模态表示学习，但主流对比方法如CLIP需要大批次大小、仔细的负采样和大量的超参数调优。我们介绍了NOVA，一种基于联合嵌入预测与分布正则化的非对比视觉语言对齐框架。NOVA通过从增强的图像视图预测文本嵌入，将视觉表示对齐到冻结的领域特定文本编码器，同时通过Sketched Isotropic Gaussian Regularization (SIGReg)强制产生各向同性高斯结构。这消除了对负采样、动量编码器或停止梯度的需要，将训练目标简化为一个超参数。我们在使用ClinicalBERT作为文本编码器和从头训练的Vision Transformers在MIMIC-CXR上进行零样本胸片分类，评估NOVA在三个基准数据集上的零样本分类表现，结果优于多个标准基线，同时表现出显著更一致的训练运行。我们的结果表明，非对比视觉语言预训练提供了一种更简单、更稳定和更有效的替代方案。
英文摘要：
Vision-language models have transformed multimodal representation learning, yet dominant contrastive approaches like CLIP require large batch sizes, careful negative sampling, and extensive hyperparameter tuning. We introduce NOVA, a NOn-contrastive Vision-language Alignment framework based on joint embedding prediction with distributional regularization. NOVA aligns visual representations to a frozen, domain-specific text encoder by predicting text embeddings from augmented image views, while enforcing an isotropic Gaussian structure via Sketched Isotropic Gaussian Regularization (SIGReg). This eliminates the need for negative sampling, momentum encoders, or stop-gradients, reducing the training objective to a single hyperparameter. We evaluate NOVA on zeroshot chest X-ray classification using ClinicalBERT as the text encoder and Vision Transformers trained from scratch on MIMIC-CXR. On zero-shot classification across three benchmark datasets, NOVA outperforms multiple standard baselines while exhibiting substantially more consistent training runs. Our results demonstrate that non-contrastive vision-language pretraining offers a simpler, more stable, and more effective alternative to contrastive methods.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
