使用白盒 Transformer 通过结构化扩散实现掩码补全
Masked Completion via Structured Diffusion with White-Box Transformers
AI总结：
本文利用扩散、压缩与掩码补全的联系提出白盒 CRATE-MAE，实现可解释的大规模无监督表征学习，并以约三成参数取得优异图像性能。
摘要：
现代学习框架通常使用海量未标注数据训练深度神经网络，通过解决简单的预文本任务来学习表征，然后将这些表征用作下游任务的基础。这些网络是基于经验设计的；因此，它们通常不可解释，其表征也不具备结构化特征，而且其设计可能存在冗余。白盒深度网络中每一层都会显式识别并转换数据中的结构，因此提供了一种很有前景的替代方案。然而，现有白盒架构仅在分类等使用标注数据的监督场景中被证明能够大规模发挥作用。在本工作中，我们首次给出了可应用于大规模无监督表征学习的白盒设计范式实例。我们利用扩散、压缩与（掩码）补全之间的一种基础联系，推导出一种类似深度 transformer 的掩码自编码器架构，称为 CRATE-MAE，其中每一层的作用在数学上都是完全可解释的：它们将数据分布转换为结构化表征，并从结构化表征转换回数据分布。大量实证评估证实了我们的分析洞见。CRATE-MAE 在大规模图像数据集上展现出极具前景的性能，而在相同模型配置下，与标准掩码自编码器相比仅使用约 30% 的参数。CRATE-MAE 学到的表征具有显式结构，同时也包含语义含义。代码可在 https://github.com/Ma-Lab-Berkeley/CRATE 获取。
英文摘要：
Modern learning frameworks often train deep neural networks with massive amounts of unlabeled data to learn representations by solving simple pretext tasks, then use the representations as foundations for downstream tasks. These networks are empirically designed; as such, they are usually not interpretable, their representations are not structured, and their designs are potentially redundant. White-box deep networks, in which each layer explicitly identifies and transforms structures in the data, present a promising alternative. However, existing white-box architectures have only been shown to work at scale in supervised settings with labeled data, such as classification. In this work, we provide the first instantiation of the white-box design paradigm that can be applied to large-scale unsupervised representation learning. We do this by exploiting a fundamental connection between diffusion, compression, and (masked) completion, deriving a deep transformer-like masked autoencoder architecture, called CRATE-MAE, in which the role of each layer is mathematically fully interpretable: they transform the data distribution to and from a structured representation. Extensive empirical evaluations confirm our analytical insights. CRATE-MAE demonstrates highly promising performance on large-scale imagery datasets while using only ~30% of the parameters compared to the standard masked autoencoder with the same model configuration. The representations learned by CRATE-MAE have explicit structure and also contain semantic meaning. Code is available at https://github.com/Ma-Lab-Berkeley/CRATE .
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
