In recent years, data selection has emerged as a core issue for large-scale visual-language model pretraining, especially on noisy web-curated datasets. One widely adopted strategy assigns quality scores such as CLIP similarity for each sample and retains the data pairs with the highest scores. However, these approaches are agnostic of data distribution and always fail to select the most informative samples. To solve this problem, we propose a simple yet theoretically principled metric named Variance Alignment Score (VAS), which has the form $\langle Σ_{\text{test}}, Σ_i\rangle$. Here, $Σ_{\text{test}}$ represents the target (cross-)covariance matrix we aim to align, potentially based on prior knowledge, while $Σ_i$ denotes the tensor product of single or multi-modal representations for the $i$-th sample. We further design a new data selection method that maximizes the total VAS. We provide theoretical analysis in a simplified setting to demonstrate the theoretical advantage of VAS over random or other existing data selection. Experimentally, applying VAS and CLIP scores together can outperform baselines by a margin of $1.3\%$ average on 38 evaluation sets for noisy dataset DataComp and $2.5\%$ on VTAB for high-quality dataset CC12M. Additionally, our ablation study also shows visual features are better than text for calculating VAS, and the related classical experimental design methods may fail under this context.
Beat-ssl: Capturing Local ECG Morphology through Heartbeat-level Contrastive Learning with Soft Targets
School of Computing Science, University of Glasgow, UK(Glasgow大学计算机科学学院);Research Center for Smart Mechatronics, National Research and Innovation Agency, Indonesia(智能机电研究中心,国家研究与创新局);School of Health & Wellbeing, University of Glasgow, UK(Glasgow大学健康与福祉学院);Department of Information Technologies, University of Limassol, Cyprus(拉马索尔大学信息科技系)
A Deep Learning Approach for Multimodal Deception Detection
A*STAR Artificial Intelligence Initiative (A*AI), Institute of High Performance Computing(新加坡科技研究局人工智能计划(A*AI),高性能计算研究所);Centro de Investigacin en Computacin, IPN(墨西哥国立理工学院计算研究中心);Temasek Laboratories, Nanyang Technological University(南洋理工大学淡马锡实验室);School of Computer Science and Engineering, Nanyang Technological University(南洋理工大学计算机科学与工程学院)
ECG-NAT: A Self-supervised Neighborhood Attention Transformer for Multi-lead Electrocardiogram Classification
Department of Computer Engineering, University of Kurdistan(库尔德斯坦大学计算机工程系);Department of Mathematics and Operational Research, University of Mons(蒙斯大学数学与运筹学系)
Learning Multiscale Consistency for Self-supervised Electron Microscopy Instance Segmentation
Institute of Artificial Intelligence, Hefei Comprehensive National Science Center(合肥综合性国家科学中心人工智能研究院);University of Science and Technology of China(中国科学技术大学)
MusicLDM: Enhancing Novelty in Text-to-Music Generation Using Beat-Synchronous Mixup Strategies
LAION;University of California San Diego(加州大学圣迭戈分校);Mila, Quebec Artificial Intelligence Institute(Mila魁北克人工智能研究所);Université de Montréal(蒙特利尔大学);University of Surrey(萨里大学)
A Hard-to-Beat Baseline for Training-free CLIP-based Adaptation
University of Science and Technology of China(中国科学技术大学);CRIPAC & MAIS, Institute of Automation, Chinese Academy of Sciences (CASIA)(中国科学院自动化研究所CRIPAC与MAIS);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Nanjing University(南京大学)