Self-supervised learning (SSL) has become an important approach in pretraining large neural networks, enabling unprecedented scaling of model and dataset sizes. While recent advances like I-JEPA have shown promising results for Vision Transformers, adapting such methods to Convolutional Neural Networks (CNNs) presents unique challenges. In this paper, we introduce CNN-JEPA, a novel SSL method that successfully applies the joint embedding predictive architecture approach to CNNs. Our method incorporates a sparse CNN encoder to handle masked inputs, a fully convolutional predictor using depthwise separable convolutions, and an improved masking strategy. We demonstrate that CNN-JEPA outperforms I-JEPA with ViT architectures on ImageNet-100, achieving a 73.3% linear top-1 accuracy using a standard ResNet-50 encoder. Compared to other CNN-based SSL methods, CNN-JEPA requires 17-35% less training time for the same number of epochs and approaches the linear and k-NN top-1 accuracies of BYOL, SimCLR, and VICReg. Our approach offers a simpler, more efficient alternative to existing SSL methods for CNNs, requiring minimal augmentations and no separate projector network.
Wadhwani Institute for Artificial Intelligence (Wadhwani AI)（瓦德瓦尼人工智能研究所（瓦德瓦尼AI））
Chest radiograph (or Chest X-Ray, CXR) is a popular medical imaging modality that is used by radiologists across the world to diagnose heart or lung conditions. Over the last decade, Convolutional Neural Networks (CNN), have seen success in identifying pathologies in CXR images. Typically, these CNNs are pretrained on the standard ImageNet classification task, but this assumes availability of large-scale annotated datasets. In this work, we analyze the utility of pretraining on unlabeled ImageNet or Chest X-Ray (CXR) datasets using various algorithms and in multiple settings. Some findings of …
Cross4D-JEPA: Dense Cross-modal Correspondence Distillation for 4D Point Cloud Representation Learning
Nagoya University（名古屋大学）；Northeastern University（东北大学）；KC Machine Learning Lab（KC机器学习实验室）；University of Science, Vietnam National University Ho Chi Minh City（胡志明市越南国立大学理科大学）；University of California, Los Angeles（加州大学洛杉矶分校）
We apply an ensemble of modified TransBTS, nnU-Net, and a combination of both for the segmentation task of the BraTS 2021 challenge. In fact, we change the original architecture of the TransBTS model by adding Squeeze-and-Excitation blocks, an increasing number of CNN layers, replacing positional encoding in Transformer block with a learnable Multilayer Perceptron (MLP) embeddings, which makes Transformer adjustable to any input size during inference. With these modifications, we are able to largely improve TransBTS performance. Inspired by a nnU-Net framework we decided to combine it with our…
VLA-JEPA: Enhancing Vision-Language-Action Model with Latent World Model
University of Science and Technology of China（中国科学技术大学）；Zhongguancun Academy, Beijing, China（中关村学院）；Shanghai Jiao Tong University（上海交通大学）；Tsinghua University（清华大学）；Eastern Institute of Technology, Ningbo（宁波东部科技研究院）；University of Chinese Academy of Sciences（中国科学院大学）；Nankai University（南开大学）
SpectralEarth-FM: Bringing Hyperspectral Imagery into Multimodal Earth Observation Pretraining
Chair of Data Science in Earth Observation（地球观测数据科学主任）；Technical University of Munich（慕尼黑技术大学）；Remote Sensing Technology Institute（遥感技术研究所）；German Aerospace Center (DLR)（德国航空航天中心）；Department of Aerospace Engineering（航空航天工程系）；University of the Bundeswehr Munich（联邦国防军慕尼黑大学）；LEAP；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）；Univ. Grenoble Alpes（格勒诺布尔阿尔卑斯大学）；Inria（法国国家信息与自动化技术研究院）；CNRS（法国国家科学研究中心）；Grenoble INP（格勒诺布尔INP）；LJK
Intel Labs（英特尔实验室）；Microsoft Research Asia（微软亚洲研究院）；Shanghai Institute of Microsystem and Information Technology, Chinese Academy of Sciences（中国科学院上海微系统与信息技术研究所）
Self-supervised vision-and-language pretraining (VLP) aims to learn transferable multi-modal representations from large-scale image-text data and to achieve strong performances on a broad scope of vision-language tasks after finetuning. Previous mainstream VLP approaches typically adopt a two-step strategy relying on external object detectors to encode images in a multi-modal Transformer framework, which suffer from restrictive object concept space, limited image context and inefficient computation. In this paper, we propose an object-aware end-to-end VLP framework, which directly feeds image …
Dalian University of Technology（大连理工大学）；Peng Cheng Laboratory（鹏城实验室）；Tiwaki Co.,Ltd.（Tiwaki株式会社）
Existing CNNs-Based RGB-D salient object detection (SOD) networks are all required to be pretrained on the ImageNet to learn the hierarchy features which helps provide a good initialization. However, the collection and annotation of large-scale datasets are time-consuming and expensive. In this paper, we utilize self-supervised representation learning (SSL) to design two pretext tasks: the cross-modal auto-encoder and the depth-contour estimation. Our pretext tasks require only a few and unlabeled RGB-D datasets to perform pretraining, which makes the network capture rich semantic contexts and…
LeVJEPA: Efficient & Scalable Video Pretraining without the Heuristics
German Cancer Research Center（德国癌症研究中心）；German Cancer Consortium（德国癌症联盟）；Goethe University Frankfurt（法兰克福大学）；Mila（米拉研究所）；Université de Montréal（蒙特利尔大学）；Brown University（布朗大学）；Courant Institute, New York University（纽约大学柯朗研究所）；Advanced Machine Intelligence (AMI Labs)（高级机器智能实验室（AMI Labs））
Qian Xuesen Laboratory of Space Technology, China Academy of Space Technology（中国空间技术研究院钱学森空间技术实验室）；School of Information and Electronics Engineering, Beijing Institute of Technology（北京理工大学信息与电子工程学院）；School of Mathematics and Computational Science, Xiangtan University（湘潭大学数学与计算科学学院）
Deep learning methods have made significant progress in ship detection in synthetic aperture radar (SAR) images. The pretraining technique is usually adopted to support deep neural networks-based SAR ship detectors due to the scarce labeled SAR images. However, directly leveraging ImageNet pretraining is hardly to obtain a good ship detector because of different imaging perspective and geometry. In this paper, to resolve the problem of inconsistent imaging perspective between ImageNet and earth observations, we propose an optical ship detector (OSD) pretraining technique, which transfers the c…
Enhancing breast cancer detection on screening mammogram using self-supervised learning and a hybrid deep model of Swin Transformer and Convolutional Neural Network
Physical Sciences, Sunnybrook Research Institute（物理科学，阳光桥研究所在）；Medical Biophysics, University of Toronto（医学生物物理，多伦多大学）
Facebook AI Research（脸书人工智能研究院）；UNC Chapel Hill（北卡罗来纳大学教堂山分校）；Dartmouth（达特茅斯学院）
Video transformers have recently emerged as a competitive alternative to 3D CNNs for video understanding. However, due to their large number of parameters and reduced inductive biases, these models require supervised pretraining on large-scale image datasets to achieve top performance. In this paper, we empirically demonstrate that self-supervised pretraining of video transformers on video-only datasets can lead to action recognition results that are on par or better than those obtained with supervised pretraining on large-scale image datasets, even massive ones such as ImageNet-21K. Since tra…
V-JEPA Policy: Building Effective World-Action Models on Predictive Visual Latents
Tsinghua University（清华大学）；Shanghai Jiao Tong University（上海交通大学）；Fudan University（复旦大学）；University of Science and Technology of China（中国科学技术大学）；Washington University in St. Louis（圣路易斯华盛顿大学）；The Institute of Artificial Intelligence, China Telecom (TeleAI)（中国电信人工智能研究院（TeleAI））
Residual Vision Transformer (ResViT) Based Self-Supervised Learning Model for Brain Tumor Classification
Center for Diabetes Technology（糖尿病技术中心）；University of Virginia（弗吉尼亚大学）；Erciyes University（埃尔齐耶斯大学）；Biocomplexity Institute and Initiative（生物复杂性研究所与倡议项目）
When seeing a new object, humans can immediately recognize it across different retinal locations: we say that the internal object representation is invariant to translation. It is commonly believed that Convolutional Neural Networks (CNNs) are architecturally invariant to translation thanks to the convolution and/or pooling operations they are endowed with. In fact, several works have found that these networks systematically fail to recognise new objects on untrained locations. In this work we show how, even though CNNs are not 'architecturally invariant' to translation, they can indeed 'learn…
Juno: Taming Predictive Latents for Vision-Language-Action Models
University of Science and Technology of China（中国科学技术大学）；Eastern Institute of Technology, Ningbo（宁波东方理工大学）；Hangzhou Dianzi University（杭州电子科技大学）；ShanghaiTech University（上海科技大学）
RadJEPA: Radiology Encoder for Chest X-Rays via Joint Embedding Predictive Architecture
Department of Computer Science and Engineering, Indian Institute of Technology Bombay（印度理工学院孟买分校计算机科学与工程系）；Department of Biomedical Engineering, Johns Hopkins University（约翰霍普金斯大学生物医学工程系）；Koita Centre for Digital Health, Indian Institute of Technology Bombay（印度理工学院孟买分校Koita数字健康中心）
Vision Transformers versus convolutional neural networks for fine-grained orchid genus identification in a species-rich, data-poor flora: a controlled benchmark on the Orchidaceae of New Guinea
James Cook University（詹姆斯库克大学）；Southwest Papua Natural Resources Conservation Agency, Ministry of Forestry, Indonesia（印度尼西亚林业部西南巴布亚自然资源保护局）；National Research and Innovation Agency (BRIN), Republic of Indonesia（印度尼西亚共和国国家研究与创新局（BRIN））；Royal Botanic Gardens, Kew（英国皇家植物园邱园）；Australian Tropical Herbarium（澳大利亚热带植物标本馆）；Commonwealth Industrial and Scientific Research Organisation (CSIRO)（澳大利亚联邦科学与工业研究组织（CSIRO））；Australian National Herbarium（澳大利亚国家植物标本馆）
Hyperspectral images (HSI) not only have a broad macroscopic field of view but also contain rich spectral information, and the types of surface objects can be identified through spectral information, which is one of the main applications in hyperspectral image related research.In recent years, more and more deep learning methods have been proposed, among which convolutional neural networks (CNN) are the most influential. However, CNN-based methods are difficult to capture long-range dependencies, and also require a large amount of labeled data for model training.Besides, most of the self-super…
Max-Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；MIT CSAIL（麻省理工学院计算机科学与人工智能实验室）
The difficulty of annotating training data is a major obstacle to using CNNs for low-level tasks in video. Synthetic data often does not generalize to real videos, while unsupervised methods require heuristic losses. Proxy tasks can overcome these issues, and start by training a network for a task for which annotation is easier or which can be trained unsupervised. The trained network is then fine-tuned for the original task using small amounts of ground truth data. Here, we investigate frame interpolation as a proxy task for optical flow. Using real movies, we train a CNN unsupervised for tem…
GOT-JEPA: Generic Object Tracking with Model Adaptation and Occlusion Handling using Joint-Embedding Predictive Architecture
Department of Computer Science, National Yang Ming Chiao Tung University（国家阳明交通大学计算机科学系）；Research Center for Information Technology Innovation, Academia Sinica（中华学术院信息技术创新研究中心）；Microsoft AI（微软人工智能）
Booz Allen Hamilton Inc.（博思艾伦汉密尔顿公司）；US Army Research Laboratory（美国陆军研究实验室）
We address two questions for training a convolutional neural network (CNN) for hyperspectral image classification: i) is it possible to build a pre-trained network? and ii) is the pre-training effective in furthering the performance? To answer the first question, we have devised an approach that pre-trains a network on multiple source datasets that differ in their hyperspectral characteristics and fine-tunes on a target dataset. This approach effectively resolves the architectural issue that arises when transferring meaningful information between the source and the target networks. To answer t…
The Chinese University of Hong Kong（香港中文大学）；University of Sydney（悉尼大学）；Shenzhen Institutes of Advanced Technology（深圳先进技术研究院）
As the intermediate level task connecting image captioning and object detection, visual relationship detection started to catch researchers' attention because of its descriptive power and clear structure. It detects the objects and captures their pair-wise interactions with a subject-predicate-object triplet, e.g. person-ride-horse. In this paper, each visual relationship is considered as a phrase with three components. We formulate the visual relationship detection as three inter-connected recognition problems and propose a Visual Phrase guided Convolutional Neural Network (ViP-CNN) to addres…
There have been considerable debates over 2D and 3D representation learning on 3D medical images. 2D approaches could benefit from large-scale 2D pretraining, whereas they are generally weak in capturing large 3D contexts. 3D approaches are natively strong in 3D contexts, however few publicly available 3D medical dataset is large and diverse enough for universal 3D pretraining. Even for hybrid (2D + 3D) approaches, the intrinsic disadvantages within the 2D / 3D parts still exist. In this study, we bridge the gap between 2D and 3D convolutions by reinventing the 2D convolutions. We propose ACS …
Symmetry Interactive Transformer with CNN Framework for Diagnosis of Alzheimer's Disease Using Structural MRI
College of Electronics and Information Engineering, Sichuan University, China（四川大学电子信息工程学院）；Tri-Institutional Center for Translational Research in Neuroimaging and Data Science (GSU, Georgia Tech, Emory), Atlanta, USA（跨机构神经影像与数据科学转化研究中心）；College of Information and Electrical Engineering, China Agricultural University, China（中国农业大学信息与电气工程学院）；School of Biomedical Engineering and Imaging Sciences, King's College London, UK（伦敦国王学院生物医学工程与影像科学学院）
