Fine-tuning large-scale pretrained models has led to tremendous progress in well-studied modalities such as vision and NLP. However, similar gains have not been observed in many other modalities due to a lack of relevant pretrained models. In this work, we propose ORCA, a general cross-modal fine-tuning framework that extends the applicability of a single large-scale pretrained model to diverse modalities. ORCA adapts to a target task via an align-then-refine workflow: given the target input, ORCA first learns an embedding network that aligns the embedded feature distribution with the pretraining modality. The pretrained model is then fine-tuned on the embedded data to exploit the knowledge shared across modalities. Through extensive experiments, we show that ORCA obtains state-of-the-art results on 3 benchmarks containing over 60 datasets from 12 modalities, outperforming a wide range of hand-designed, AutoML, general-purpose, and task-specific methods. We highlight the importance of data alignment via a series of ablation studies and demonstrate ORCA's utility in data-limited regimes.
IsoCLIP: Decomposing CLIP Projectors for Efficient Intra-modal Alignment
Media Integration and Communication Center (MICC), University of Florence, Italy(意大利佛罗伦萨大学媒体集成与通信中心);Department of Computer Science, Universitat Autònoma de Barcelona, Spain(西班牙巴塞罗那自治大学计算机科学系);Computer Vision Center, Barcelona, Spain(西班牙巴塞罗那计算机视觉中心);IDEAS Research Institute, Warsaw, Poland(波兰华沙IDEAS研究所)
OVA-IB: One vs All Information Bottleneck for Multi-Modal Alignment
Hong Kong University of Science and Technology(香港科学与技术大学);Vrije Universiteit Amsterdam(阿姆斯特丹自由大学);UiT – The Arctic University of Norway(挪威北极大学);University of Copenhagen(哥本哈根大学);Norwegian Computing Center(挪威计算中心);University of Massachusetts Amherst(马萨诸塞大学阿姆赫斯特分校)
Cross-modal Affinity-aligned Multimodal Learning Analytics for Predicting Student Collaboration Satisfaction in Game-Based Learning
Institute of Education, National Cheng Kung University(国立成功大学教育研究所);Institute of Intelligent System, National Yang Ming Chiao Tung University(阳明交通大学智能系统研究所);Department of Computer Science, University at Albany, State University of New York(纽约州立大学水牛城分校计算机科学系)
From Consistency to Complementarity: Aligned and Disentangled Multi-modal Learning for Time Series Understanding and Reasoning
The Hong Kong University of Science(香港科学与技术大学);Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);University of Chinese Academy of Sciences(中国科学院大学)