用于领域无关自监督学习的自引导掩码自编码器
Self-Guided Masked Autoencoders for Domain-Agnostic Self-Supervised Learning
AI总结：
本文提出自引导掩码自编码器SMA，通过可学习的掩码采样消除领域特定假设，实现完全领域无关的自监督学习，并在蛋白质、化学和粒子物理三个基准上取得最优性能。
摘要：
自监督学习擅长从大量无标签数据中学习表示，已在多种数据模态上展现出成功。然而，将自监督学习扩展到新模态并非易事，因为现有方法的具体设计均针对各自领域定制，例如反映目标任务不变性的领域特定数据增强。尽管掩码建模作为一种领域无关的自监督学习框架具有前景，因为它不依赖输入增强，但其掩码采样过程仍具有领域特定性。我们提出自引导掩码自编码器（Self-guided Masked Autoencoders，SMA），一种完全领域无关的掩码建模方法。SMA 通过掩码建模目标训练基于注意力的模型，在不依赖任何领域特定假设的情况下学习采样掩码。我们在蛋白质生物学、化学性质预测和粒子物理学三个自监督学习基准上评估 SMA。我们发现 SMA 能够在没有领域特定知识的情况下学习表示，并在这三个基准上取得了最先进的性能。
英文摘要：
Self-supervised learning excels in learning representations from large amounts of unlabeled data, demonstrating success across multiple data modalities. Yet, extending self-supervised learning to new modalities is non-trivial because the specifics of existing methods are tailored to each domain, such as domain-specific augmentations which reflect the invariances in the target task. While masked modeling is promising as a domain-agnostic framework for self-supervised learning because it does not rely on input augmentations, its mask sampling procedure remains domain-specific. We present Self-guided Masked Autoencoders (SMA), a fully domain-agnostic masked modeling method. SMA trains an attention based model using a masked modeling objective, by learning masks to sample without any domain-specific assumptions. We evaluate SMA on three self-supervised learning benchmarks in protein biology, chemical property prediction, and particle physics. We find SMA is capable of learning representations without domain-specific knowledge and achieves state-of-the-art performance on these three benchmarks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
