Combining Reconstruction and Contrastive Methods for Multimodal Representations in RL
Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）；FZI Research Center for Information Technology（FZI信息技术研究中心）；Bosch Center for Artificial Intelligence（博世人工智能中心）；University of Tübingen（蒂宾根大学）
Philipp Becker, Sebastian Mossburger, Fabian Otto, Gerhard Neumann
备注：
Published in "Reinforcement Learning Conference (RLC)", August 2024
Learning self-supervised representations using reconstruction or contrastive losses improves performance and sample complexity of image-based and multimodal reinforcement learning (RL). Here, different self-supervised loss functions have distinct advantages and limitations depending on the information density of the underlying sensor modality. Reconstruction provides strong learning signals but is susceptible to distractions and spurious information. While contrastive approaches can ignore those, they may fail to capture all relevant details and can lead to representation collapse. For multimodal RL, this suggests that different modalities should be treated differently based on the amount of distractions in the signal. We propose Contrastive Reconstructive Aggregated representation Learning (CoRAL), a unified framework enabling us to choose the most appropriate self-supervised loss for each sensor modality and allowing the representation to better focus on relevant aspects. We evaluate CoRAL's benefits on a wide range of tasks with images containing distractions or occlusions, a new locomotion suite, and a challenging manipulation suite with visually realistic distractions. Our results show that learning a multimodal representation by combining contrastive and reconstruction-based losses can significantly improve performance and solve tasks that are out of reach for more naive representation learning approaches and other recent baselines.
Orthogonalized Multimodal Contrastive Learning with Asymmetric Masking for Structured Representations
Peter L. Reichertz Institute for Medical Informatics of TU Braunschweig and Hannover Medical School（图林根工业大学和汉诺威医学院医学信息学研究所）；Lower Saxony Center for AI and Causal Methods in Medicine (CAIMed)（下萨克森人工智能与因果医学中心（CAIMed））
Hidden in the Multiplicative Interaction: Uncovering Fragility in Multimodal Contrastive Learning
Berlin Institute of Health, Charité - Universitätsmedizin Berlin（柏林健康研究所，柏林查理医院）；Intelligent Medicine Institute, Fudan University（智能医学研究院，复旦大学）；Department of Mathematics and Computer Science, Freie Universität Berlin（数学与计算机科学系，柏林自由大学）
Beyond Objective Expressivity: Geometry Preservation in Multimodal Contrastive Learning
Berlin Institute of Health, Charité - Universitätsmedizin Berlin（柏林健康研究院，柏林夏里特大学医学中心）；Department of Mathematics and Computer Science, Freie Universität Berlin（柏林自由大学数学与计算机科学系）；Intelligent Medicine Institute, Fudan University（复旦大学智能医学研究院）
Contrastive Learning on Multimodal Analysis of Electronic Health Records
Harvard T.H. Chan School of Public Health（哈佛T.H. 柏林公共卫生学院）；Harvard Medical School（哈佛医学院）；Rutgers University（罗格斯大学）；National University of Singapore（新加坡国立大学）
Prior-Fitted and Frozen Scorers for Multimodal Contrastive Learning
Berlin Institute of Health（柏林卫生研究所）；Charité - Universitätsmedizin Berlin（柏林夏里特医学院）；Freie Universität Berlin（柏林自由大学）；Intelligent Medicine Institute（智能医学研究所）；Fudan University（复旦大学）
Local-Global Multimodal Contrastive Learning for Molecular Property Prediction
School of Mathematical Sciences（数学科学学院）；University of Electronic Science and Technology of China（电子科技大学）；Department of Computer Science and Engineer（计算机科学与工程系）；Oakland University（奥克兰大学）；Department of Electrical and Computer Engineering（电气与计算机工程系）；College of Management Science（管理科学学院）；Chengdu University of Technology（成都理工大学）
MVCL-DAF++: Enhancing Multimodal Intent Recognition via Prototype-Aware Contrastive Alignment and Coarse-to-Fine Dynamic Attention Fusion
University of Shanghai for Science and TechnologyChina（上海科学技术大学中国）；Shenzhen Institute of Advanced Technology, Chinese Academy of SciencesChina（深圳先进技术研究院，中国科学院中国）；University of Minnesota-Twin Cities, USA（明尼苏达大学双城分校，美国）；University of LeedsUK（利兹大学，英国）
Multimodal Federated Learning via Contrastive Representation Ensemble
Institute for AI Industry Research, Tsinghua University（清华大学人工智能产业研究院）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；University of Waterloo（滑铁卢大学）；Carnegie Mellon University（卡内基梅隆大学）
Token-Level Contrastive Learning with Modality-Aware Prompting for Multimodal Intent Recognition
Tsinghua University（清华大学）；Beijing National Research Center for Information Science and Technology（北京信息科学与技术国家研究中心）；Hebei University of Science and Technology（河北科技大学）
Towards Fine-grained Visual Representations by Combining Contrastive Learning with Image Reconstruction and Attention-weighted Pooling
Bayer AG（拜耳股份公司）
本文介绍了对比重建，ConRec -一种自监督学习算法，通过联合优化对比和自重建损失来获得图像表示。我们展示了最先进的对比学习方法（例如Simplified）在其表示中捕获细粒度视觉特征方面存在缺陷。ConRec通过添加（1）自我重建任务和（2）对比学习任务中的注意力机制扩展了Simplified框架。这是通过应用具有两个头的简单编码器-解码器架构来实现的。我们表明，这两个扩展有助于改善图像的矢量表示与细粒度的视觉特征。结合这些概念，ConRec在细粒度分类数据集上的性能优于Simplified和Simplified with Attention-Pooling。
CAMEL-CLIP: Channel-aware Multimodal Electroencephalography-text Alignment for Generalizable Brain Foundation Models
Department of Computer Science and Engineering（计算机科学与工程系）；Incheon National University（庆尚国立大学）；Department of Artificial Intelligence（人工智能系）；Inha University（Inha大学）；Center for Brain-Machine Interface（脑机接口中心）
An Enhanced Dual Transformer Contrastive Network for Multimodal Sentiment Analysis
Ho Chi Minh City Open University, Ho Chi Minh, Vietnam Insight Centre for Data Analytics \& School of Computing, Dublin City University, Dublin, Ireland
