Joint Embedding Predictive Architectures (JEPAs) learn representations able to solve numerous downstream tasks out-of-the-box. JEPAs combine two objectives: (i) a latent-space prediction term, i.e., the representation of a slightly perturbed sample must be predictable from the original sample's representation, and (ii) an anti-collapse term, i.e., not all samples should have the same representation. While (ii) is often considered as an obvious remedy to representation collapse, we uncover that JEPAs' anti-collapse term does much more--it provably estimates the data density. In short, any successfully trained JEPA can be used to get sample probabilities, e.g., for data curation, outlier detection, or simply for density estimation. Our theoretical finding is agnostic of the dataset and architecture used--in any case one can compute the learned probabilities of sample $x$ efficiently and in closed-form using the model's Jacobian matrix at $x$. Our findings are empirically validated across datasets (synthetic, controlled, and Imagenet) and across different Self Supervised Learning methods falling under the JEPA family (I-JEPA and DINOv2) and on multimodal models, such as MetaCLIP. We denote the method extracting the JEPA learned density as {\bf JEPA-SCORE}.
US-JEPA: A Joint Embedding Predictive Architecture for Ultrasound
Biomedical AI Research Lab, University of California, Los Angeles, USA(生物医学人工智能研究实验室,加州大学洛杉矶分校);Department of Bioengineering, University of California, Los Angeles, USA(生物工程系,加州大学洛杉矶分校);Medical Informatics Home Area, University of California, Los Angeles, USA(医学信息学家庭区域,加州大学洛杉矶分校);Department of Electrical and Computer Engineering, University of California, Los Angeles, USA(电气与计算机工程系,加州大学洛杉矶分校);Department of Pathology, University of California, Los Angeles, USA(病理学系,加州大学洛杉矶分校);Department of Radiology, University of California, Los Angeles, USA(放射学系,加州大学洛杉矶分校)
MC-JEPA: A Joint-Embedding Predictive Architecture for Self-Supervised Learning of Motion and Content Features
Meta AI;Inria(法国国家信息与自动化研究所);École normale supérieure(巴黎高等师范学院);CNRS(法国国家科学研究中心);PSL Research University(巴黎文理研究大学);Courant Institute, New York University(纽约大学柯朗研究所);Center for Data Science, New York University(纽约大学数据科学中心)
Your Contrastive Learning Is Secretly Doing Stochastic Neighbor Embedding
Huawei Noah’s Ark Lab(华为诺亚方舟实验室);Hong Kong University of Science and Technology(香港科技大学);Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));Qing Yuan Research Institute, Shanghai Jiao Tong University(上海交通大学清源研究院)
Graph-structured joint-embedding prediction for single-cell transcriptomics
Westlake University(西湖大学);Centre for Artificial Intelligence and Robotics (CAIR), Hong Kong Institute of Science & Innovation, Chinese Academy of Sciences (HKISI-CAS)(中国科学院香港创新研究院人工智能与机器人中心);Institute of Automation, Chinese Academy of Sciences (CASIA)(中国科学院自动化研究所);University of Chinese Academy of Sciences (UCAS)(中国科学院大学)
SPHERE-JEPA: Spherical Prediction with Homogeneous Embeddings
Université Paris-Saclay, CNRS, Advanced Track and Trace(巴黎萨克雷大学,国家科学研究中心,先进跟踪与追溯);ENS Paris-Saclay, Centre Borelli, Advanced Track and Trace(巴黎萨克雷高等师范学院,博雷利中心,先进跟踪与追溯);Université Paris-Saclay, CNRS(巴黎萨克雷大学,国家科学研究中心);ENS Paris-Saclay, Centre Borelli(巴黎萨克雷高等师范学院,博雷利中心)
Delta-JEPA: Learning Action-Sensitive World Models via Latent Difference Decoding
School of Computer Science and Technology, University of Chinese Academy of Sciences(中国科学院大学计算机科学与技术学院);Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Computer Science and Technology, Harbin Institute of Technology, Weihai(哈尔滨工业大学(威海)计算机科学与技术学院);Faculty of Computing, Harbin Institute of Technology, Harbin(哈尔滨工业大学计算学部);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所)
JetParticle-JEPA: An Efficient Self-Supervised Representation Learning method for Jet Tagging in High-Energy Physics
GREYC, Normandy University, Unicaen, ENSICAEN, UMR CNRS 6072(GREYC,诺曼底大学,Unicaen,ENSICAEN,CNRS UMR 6072);LPC, Normandy University, Unicaen, ENSICAEN, IN2P3, UMR CNRS 6534(LPC,诺曼底大学,Unicaen,ENSICAEN,IN2P3,CNRS UMR 6534)
Soft Clustering Anchors for Self-Supervised Speech Representation Learning in Joint Embedding Prediction Architectures
Carnegie Mellon University(卡内基梅隆大学);New York University(纽约大学);James Silberrad Brown Center for AI(詹姆斯·西伯拉德·布朗人工智能中心);Columbia University(哥伦比亚大学);Northeastern University(东北大学);Stanford University(斯坦福大学);Amazon GenAI(亚马逊生成人工智能)
JEPA as a Neural Tokenizer: Learning Robust Speech Representations with Density Adaptive Attention
Carnegie Mellon University(卡内基梅隆大学);Amazon GenAI(亚马逊生成人工智能);James Silberrad Brown Center for Artificial Intelligence(詹姆斯·西伯拉德·布朗人工智能中心);University of Bristol(布里斯托大学);Stanford University(斯坦福大学);Northeastern University(东北大学);New York University(纽约大学)
CF-JEPA: Mask-free forward prediction with asymmetric encoder utilization for time-series representation learning
Graduate School of Artificial Intelligence Convergence Engineering, Changwon National University(昌原国立大学人工智能融合工程研究生院);Department of Artificial Intelligence Engineering, Changwon National University(昌原国立大学人工智能工程系)