The Platonic Representation Hypothesis posits that learned representations from models trained on different modalities converge to a shared latent structure of the world. However, this hypothesis has largely been examined in vision and language, and it remains unclear whether time series participate in such convergence. We first examine this in a trimodal setting and find that independently pretrained time series, vision, and language encoders exhibit near-orthogonal geometry in the absence of explicit coupling. We then apply post-hoc alignment by training projection heads over frozen encoders using contrastive learning, and analyze the resulting representations with respect to geometry, scaling behavior, and dependence on information density and input modality characteristics. Our investigation reveals that overall alignment in contrastive representation spaces improves with model size, but this alignment is asymmetric: time series align more strongly with visual representations than with text, and images can act as effective intermediaries between time series and language. We further see that richer textual descriptions improve alignment only up to a threshold; training on denser captions does not lead to further improvement. Analogous effects are observed for visual representations. Our findings shed light on considerations for building multimodal systems involving non-conventional data modalities beyond vision and language.
CARE: Contrastive Alignment for ADL Recognition from Event-Triggered Sensor Streams
Electrical and Computer Engineering University of Maryland, College Park（马里兰大学电气与计算机工程学院）；School of Computing University of Georgia（佐治亚大学计算机科学学院）；Eindhoven University of Technology（埃因霍温理工大学）
University of Science and Technology of China（中国科学技术大学）；JD Explore Academy（京东探索研究院）；The University of Sydney（悉尼大学）
Data-Efficient GANs (DE-GANs), which aim to learn generative models with a limited amount of training data, encounter several challenges for generating high-quality samples. Since data augmentation strategies have largely alleviated the training instability, how to further improve the generative performance of DE-GANs becomes a hotspot. Recently, contrastive learning has shown the great potential of increasing the synthesis quality of DE-GANs, yet related principles are not well explored. In this paper, we revisit and compare different contrastive learning strategies in DE-GANs, and identify (…
Exploring the Potential of Contrastive Language-Image Pre-training for Multi-Source Remote Sensing Data
School of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；Space-based Computing System Research Center, Zhejiang Lab（之江实验室天基计算系统研究中心）
Microsoft Research（微软研究院）；Carnegie Mellon University（卡内基梅隆大学）
Contrastive learning has been widely used to train transformer-based vision-language models for video-text alignment and multi-modal representation learning. This paper presents a new algorithm called Token-Aware Cascade contrastive learning (TACo) that improves contrastive learning using two novel techniques. The first is the token-aware contrastive loss which is computed by taking into account the syntactic classes of words. This is motivated by the observation that for a video-text pair, the content words in the text, such as nouns and verbs, are more likely to be aligned with the visual co…
Contrastive self-supervised learning (CSL) with a prototypical regularization has been introduced in learning meaningful representations for downstream tasks that require strong semantic information. However, to optimize CSL with a loss that performs the prototypical regularization aggressively, e.g., the ProtoNCE loss, might cause the "coagulation" of examples in the embedding space. That is, the intra-prototype diversity of samples collapses to trivial solutions for their prototype being well-separated from others. Motivated by previous works, we propose to mitigate this phenomenon by learni…
SenseTime Research（商汤科技研究院）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；University of California, Los Angeles（加州大学洛杉矶分校）
The essence of unsupervised anomaly detection is to learn the compact distribution of normal samples and detect outliers as anomalies in testing. Meanwhile, the anomalies in real-world are usually subtle and fine-grained in a high-resolution image especially for industrial applications. Towards this end, we propose a novel framework for unsupervised anomaly detection and localization. Our method aims at learning dense and compact distribution from normal images with a coarse-to-fine alignment process. The coarse alignment stage standardizes the pixel-wise position of objects in both image and …
Guangdong University of Technology（广东工业大学）；The University of Sydney（悉尼大学）
Unpaired image-to-image translation aims to find a mapping between the source domain and the target domain. To alleviate the problem of the lack of supervised labels for the source images, cycle-consistency based methods have been proposed for image structure preservation by assuming a reversible relationship between unpaired images. However, this assumption only uses limited correspondence between image pairs. Recently, contrastive learning (CL) has been used to further investigate the image correspondence in unpaired image translation by using patch-based positive/negative learning. Patch-ba…
Multi-modal entity alignment aims to identify equivalent entities between two different multi-modal knowledge graphs, which consist of structural triples and images associated with entities. Most previous works focus on how to utilize and encode information from different modalities, while it is not trivial to leverage multi-modal knowledge in entity alignment because of the modality heterogeneity. In this paper, we propose MCLEA, a Multi-modal Contrastive Learning based Entity Alignment model, to obtain effective joint representations for multi-modal entity alignment. Different from previous …
Tsinghua University（清华大学）；Beijing University of Chemical Technology（北京化工大学）；Singapore Management University（新加坡管理大学）；Information Research Center of Military Science（军事科学院信息研究中心）；China Mobile Research Institute（中国移动研究院）
Self-supervised entity alignment (EA) aims to link equivalent entities across different knowledge graphs (KGs) without seed alignments. The current SOTA self-supervised EA method draws inspiration from contrastive learning, originally designed in computer vision based on instance discrimination and contrastive loss, and suffers from two shortcomings. Firstly, it puts unidirectional emphasis on pushing sampled negative entities far away rather than pulling positively aligned pairs close, as is done in the well-established supervised EA. Secondly, KGs contain rich side information (e.g., entity …
Peking University（北京大学）；The University of Sydney（悉尼大学）；iFlytek Research（讯飞研究院）
Word alignment is essential for the downstream cross-lingual language understanding and generation tasks. Recently, the performance of the neural word alignment models has exceeded that of statistical models. However, they heavily rely on sophisticated translation models. In this study, we propose a super lightweight unsupervised word alignment model named MirrorAlign, in which bidirectional symmetric attention trained with a contrastive learning objective is introduced, and an agreement loss is employed to bind the attention maps, such that the alignments follow mirror-like symmetry hypothesi…
CLHA: A Simple yet Effective Contrastive Learning Framework for Human Alignment
University of Science and Technology of China（中国科学技术大学）；Southern University of Science and Technology（南方科技大学）；Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；Shenzhen MSU-BIT University（深圳北理莫斯科大学）；Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳））
为解决RLHF训练复杂困难的问题，提出CLHA框架，通过重新评分策略动态调整训练过程，并结合成对对比损失与自适应监督微调损失直接对齐LLMs与人类偏好，在“Helpful and Harmless”数据集上表现卓越。
Perfect Alignment May be Poisonous to Graph Contrastive Learning
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学高瓴人工智能学院）；Beijing Key Laboratory of Big Data Management and Analysis Methods（北京大数据管理与分析方法重点实验室）
Automated Human Activity Recognition has long been a problem of great interest in human-centered and ubiquitous computing. In the last years, a plethora of supervised learning algorithms based on deep neural networks has been suggested to address this problem using various modalities. While every modality has its own limitations, there is one common challenge. Namely, supervised learning requires vast amounts of annotated data which is practically hard to collect. In this paper, we benefit from the self-supervised learning paradigm (SSL) that is typically used to learn deep feature representat…
Personalized Federated Learning for Spatio-Temporal Forecasting: A Dual Semantic Alignment-Based Contrastive Approach
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））
Despite the great success of spoken language understanding (SLU) in high-resource languages, it remains challenging in low-resource languages mainly due to the lack of labeled training data. The recent multilingual code-switching approach achieves better alignments of model representations across languages by constructing a mixed-language context in zero-shot cross-lingual SLU. However, current code-switching methods are limited to implicit alignment and disregard the inherent semantic structure in SLU, i.e., the hierarchical inclusion of utterances, slots, and words. In this paper, we propose…
Machine Unlearning in Hyperbolic vs. Euclidean Multimodal Contrastive Learning: Adapting Alignment Calibration to MERU
Aalborg University（奥尔堡大学）；University of Barcelona（巴塞罗那大学）；Computer Vision Center（计算机视觉中心）；Milestone Systems（里程碑系统公司）；Pioneer Center for Artificial Intelligence（先锋人工智能中心）
Aligning image and text encoders from scratch using contrastive learning requires large amounts of paired image-text data. We alleviate this need by aligning individually pre-trained language and vision representation models using a much smaller amount of paired data, augmented with a curriculum learning algorithm to learn fine-grained vision-language alignments. TOnICS (Training with Ontology-Informed Contrastive Sampling) initially samples minibatches whose image-text pairs contain a wide variety of objects to learn object-level alignment, and progressively samples minibatches where all imag…
Cross-Subject EEG Emotion Recognition Based on Temporal Asynchronous Alignment Contrastive Learning
School of Biomedical Engineering, Shenzhen Campus of Sun Yat-sen University（中山大学生物医学工程学院深圳校区）；School of Computer Science and Technology, Tianjin University（天津大学计算机科学与技术学院）
Meituan（美团）；Ping An Technology (Shenzhen) Co., Ltd（平安科技（深圳）有限公司）；University of Nottingham Ningbo China（宁波诺丁汉大学）
Integrating multimodal knowledge for abstractive summarization task is a work-in-progress research area, with present techniques inheriting fusion-then-generation paradigm. Due to semantic gaps between computer vision and natural language processing, current methods often treat multiple data points as separate objects and rely on attention mechanisms to search for connection in order to fuse together. In addition, missing awareness of cross-modal matching from many frameworks leads to performance reduction. To solve these two drawbacks, we propose an Iterative Contrastive Alignment Framework (…
CLAR: Learning 3D Representations for Robotic Manipulation by Fusing Masked Reconstruction with Multi-Level Contrastive Alignment
SKL-MAIS, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所SKL-MAIS）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Carnegie Mellon University（卡内基梅隆大学）；Galbot；CFCS, School of Computer Science, Peking University（北京大学计算机科学与技术学院CFCS）
Seeing the Image: Prioritizing Visual Correlation by Contrastive Alignment
School of Computer Science, Wuhan University（武汉大学计算机学院）；ByteDance Inc.（字节跳动公司）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）
Defending Against Backdoor Attacks via Alignment Checking in Model-Contrastive Federated Learning
Shandong Computer Science Center（山东计算机科学中心）；Qilu University of Technology (Shandong Academy of Sciences)（齐鲁工业大学（山东省科学院））；College of computer science and technology, China University of Petroleum（中国石油大学计算机科学与技术学院）；School of Computer Science and Engineering, University of Electronic Science and Technology of China（电子科技大学计算机科学与工程学院）；Big Data Institute, Qilu University of Technology（齐鲁工业大学大数据研究院）
MVCL-DAF++: Enhancing Multimodal Intent Recognition via Prototype-Aware Contrastive Alignment and Coarse-to-Fine Dynamic Attention Fusion
University of Shanghai for Science and TechnologyChina（上海科学技术大学中国）；Shenzhen Institute of Advanced Technology, Chinese Academy of SciencesChina（深圳先进技术研究院，中国科学院中国）；University of Minnesota-Twin Cities, USA（明尼苏达大学双城分校，美国）；University of LeedsUK（利兹大学，英国）
