The lack of quality labeled data is one of the main bottlenecks for training Deep Learning models. As the task increases in complexity, there is a higher penalty for overfitting and unstable learning. The typical paradigm employed today is Self-Supervised learning, where the model attempts to learn from a large corpus of unstructured and unlabeled data and then transfer that knowledge to the required task. Some notable examples of self-supervision in other modalities are BERT for Large Language Models, Wav2Vec for Speech Recognition, and the Masked AutoEncoder for Vision, which all utilize Transformers to solve a masked prediction task. GeoAI is uniquely poised to take advantage of the self-supervised methodology due to the decades of data collected, little of which is precisely and dependably annotated. Our goal is to extract building and road segmentations from Digital Elevation Models (DEM) that provide a detailed topography of the earths surface. The proposed architecture is the Masked Autoencoder pre-trained on ImageNet (with the limitation that there is a large domain discrepancy between ImageNet and DEM) with an UperNet Head for decoding segmentations. We tested this model with 450 and 50 training images only, utilizing roughly 5% and 0.5% of the original data respectively. On the building segmentation task, this model obtains an 82.1% Intersection over Union (IoU) with 450 Images and 69.1% IoU with only 50 images. On the more challenging road detection task the model obtains an 82.7% IoU with 450 images and 73.2% IoU with only 50 images. Any hand-labeled dataset made today about the earths surface will be immediately obsolete due to the constantly changing nature of the landscape. This motivates the clear necessity for data-efficient learners that can be used for a wide variety of downstream tasks.
Advancing ALS Applications with Large-Scale Pre-training: Dataset Development and Downstream Assessment
organization= National Institute of Advanced Industrial Science;Technology (AIST) Artificial Intelligence Research Center , city= Tokyo , country= Japan
Aalborg University(奥尔堡大学);University Politehnica of Bucharest(布加勒斯特理工大学);University of Bucharest(布加勒斯特大学);Milestone Systems(迈尔斯通系统公司);SecurifAI(SecurifAI公司)
本文提出 CL-MAE,利用可学习掩码模块在 MAE 自监督训练中生成由易到难的重建课程,并在 ImageNet 及五个下游任务上验证其优于 MAE 的表示学习效果。
PT-Tuning: Bridging the Gap between Time Series Masked Reconstruction and Forecasting via Prompt Token Tuning
State Grid Smart Grid Research Institute Co.Ltd.(国网智能电网研究院有限公司);Wuhan Nari Limited Liability Company of State Grid Electric Power Research Institute(国网电力科学研究院武汉南瑞有限责任公司);State Grid Inner Mongolia East Power Co., Ltd(国网内蒙古东部电力有限公司);Electric Power Research Institute of State Grid Anhui Electric Power Co., Ltd(国网安徽电力科学研究院);State Grid Anhui Ultra High Voltage Company(国网安徽超高压公司)
Contrastive Tuning: A Little Help to Make Masked Autoencoders Forget
Faculty of Computer Science, University of Vienna, Vienna, Austria(维也纳大学计算机科学学院);UniVie Doctoral School Computer Science, University of Vienna(维也纳大学计算机科学博士学院);Institute of Advanced Research in Artificial Intelligence (IARAI)(高级人工智能研究所)
提出 MAE-CT 方法,结合 MIM 的效率与 ID 的对象聚焦能力,通过 NNCLR 目标在预训练 MAE 顶层诱导语义聚类,无需标签且计算开销低,在 ViT 模型上多项指标达 SOTA。
LAVENDER: Unifying Video-Language Understanding as Masked Language Modeling
Microsoft(微软)
近年来,统一视觉语言框架有了很大的发展,其中大多数采用编码器-解码器架构,将图像-文本任务统一为序列到序列生成。然而,现有的视频语言(VidL)模型仍然需要在模型架构和每个任务的训练目标的任务特定的设计。在这项工作中,我们探索了一个统一的VidL框架LAVENDER,其中Masked Language Modeling(MLM)被用作所有预训练和下游任务的公共接口。这样的统一导致简化的模型架构,其中在多模态编码器之上仅需要轻量级MLM头,而不是具有更多参数的解码器。令人惊讶的是,实验结果表明,这个统一的框架实现了竞争力的性能在14个VidL基准,涵盖视频问答,文本到视频检索和视频描述。广泛的分析进一步证明了LAVENDER优于现有VidL方法的优点:(i)在多任务微调时仅使用一组参数值支持所有下游任务;(ii)对各种下游任务的Few-Shot泛化;以及(iii)对视频问答任务启用zero-shot评估。代码可在https://github.com/microsoft/LAVENDER上获得。
MLIM: Vision-and-Language Model Pre-training with Masked Language and Image Modeling
Amazon.com Inc(亚马逊公司)
视觉和语言预训练(VLP)可提高需要图像和文本输入的下游任务的模型性能。当前的VLP方法在(i)模型架构(特别是图像嵌入器)、(ii)损失函数和(iii)掩蔽策略上不同。图像嵌入器要么是像ResNet这样的深度模型,要么是直接将图像像素馈送到Transformer中的线性投影。通常,除了掩蔽语言建模(MLM)损失之外,基于语义的目标用于跨模态交互,RoI特征回归和分类任务用于掩蔽图像区域建模(MLM)。对齐和MPEG4目标大多没有地面实况。基于对齐的目标需要图像和文本的配对以及启发式目标函数。机器人依靠物体探测器。掩蔽策略要么不利用多模态,要么与其他模型生成的对齐严格耦合。在本文中,我们提出了掩蔽语言和图像建模(MLIM)的VLP。MLIM使用两个损失函数:Masked Language Modeling(MLM)损失和图像重建(RECON)损失。我们提出了模态感知掩蔽(MAM)来增强跨模态交互,并利用MLM和RECON损失分别捕获文本和图像重建质量。使用MLM + RECON任务加上MAM,我们提出了一个简化的VLP方法,并表明它具有更好的下游任务性能的专有电子商务多模态数据集。
Fine-tuning a Multiple Instance Learning Feature Extractor with Masked Context Modelling and Knowledge Distillation
Institute for Biomedical Informatics, Faculty of Medicine(医学院生物医学信息学研究所);University Hospital Cologne, University of Cologne(科隆大学医院(科隆大学));Center for Molecular Medicine Cologne (CMMC), Faculty of Medicine(科隆分子医学中心(CMMC,医学院))
MaeFuse: Transferring Omni Features with Pretrained Masked Autoencoders for Infrared and Visible Image Fusion via Guided Training
School of Computer Science and Technology, Harbin Institute of Technology(哈尔滨工业大学计算机科学与技术学院);Electronic Information School, Wuhan University(武汉大学电子信息学院);School of Computer Science and Technology, Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳)计算机科学与技术学院)
Deblurring Masked Autoencoder is Better Recipe for Ultrasound Image Recognition
West China Hospital, Sichuan University(四川大学华西医院);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Sichuan University(四川大学);Beijing University of Posts and Telecommunications(北京邮电大学)
MST: Masked Self-Supervised Transformer for Visual Representation
Institute of Automation, CAS(中国科学院自动化研究所);University of Chinese Academy of Sciences(中国科学院大学);SenseTime Research(商汤科技研究院);University of California, Los Angeles(加利福尼亚大学洛杉矶分校)
Transformer已被广泛应用于自然语言处理(NLP)中的自监督预训练,并取得了巨大的成功。然而,在视觉自监督学习中,它还没有得到充分的探索。同时,以往的方法只从全局的角度考虑高层特征和学习表示,可能无法将其转移到关注局部特征的下游密集预测任务中。在本文中,我们提出了一种新的Masked自监督Transformer方法命名为MST,它可以显式地捕获图像的局部上下文,同时保留全局语义信息。具体而言,受自然语言处理中的MLM(Masked Language Modeling)思想的启发,提出了一种基于多头自注意映射的Masked token策略,该策略在不破坏自监督学习关键结构的前提下,动态地屏蔽局部块的部分token。更重要的是,屏蔽令牌连同剩余令牌一起由全局图像解码器进一步恢复,该全局图像解码器保留了图像的空间信息,并且对下游密集预测任务更友好。在多个数据集上的实验证明了该方法的有效性和通用性。例如,MST使用DeiT-S仅使用300个epoch预训练通过线性评估实现了76.9%的Top-1准确率,这比具有相同epoch的监督方法高出0.4%,比其可比变体DINO高出1.0%。对于密集预测任务,MST还在MS COCO目标检测上实现了42.7%的mAP,在Cityscapes分割上实现了74.04%的mIoU,仅使用100个epoch的预训练。
Hi-End-MAE: Hierarchical encoder-driven masked autoencoders are stronger vision learners for medical image segmentation
University of Science and Technology of China (USTC)(中国科学技术大学);Suzhou Institute for Advanced Research, USTC(中国科学技术大学苏州高等研究院);Stanford University(斯坦福大学);State Key Laboratory of Precision and Intelligent Chemistry, USTC(中国科学技术大学精准与智能化学国家重点实验室);Key Laboratory of Intelligent Information Processing of Chinese Academy of Sciences (CAS), Institute of Computing Technology, CAS(中国科学院计算技术研究所中国科学院智能信息处理重点实验室)