基于注意力的神经元胞自动机
Attention-based Neural Cellular Automata
摘要：
细胞自动机（CA）的最新扩展已经融入了现代深度学习的关键思想，极大地扩展了它们的功能，并催生了一系列新的神经细胞自动机（NCA）技术。受基于transformer的架构的启发，我们的工作提出了一类新的$\textit{attention-based}$ NCA形成使用空间本地化的$\unicode{x2014}$，但全球组织的$\unicode{x2014}$自我注意力计划。我们引入一个名为$\textit{Vision Transformer Cellular Automata}$（ViTCA）的该类实例。我们提供了六个基准数据集的去噪自动编码的定量和定性结果，将ViTCA与U-Net、基于U-Net的CA基线（UNetCA）和视觉Transformer（ViT）进行比较。当比较配置为类似参数复杂性的架构时，ViTCA架构在所有基准测试和几乎每个评估指标中都具有卓越的性能。我们提出了一个烧蚀研究的各种建筑配置的ViTCA，分析其对细胞状态的影响，并调查其感应偏置。最后，我们通过线性探针对其收敛的单元状态隐藏表示进行了检查，与我们的U-Net，ViT和UNetCA基线相比，平均而言，结果更好。
英文摘要：
Recent extensions of Cellular Automata (CA) have incorporated key ideas from modern deep learning, dramatically extending their capabilities and catalyzing a new family of Neural Cellular Automata (NCA) techniques. Inspired by Transformer-based architectures, our work presents a new class of $\textit{attention-based}$ NCAs formed using a spatially localized$\unicode{x2014}$yet globally organized$\unicode{x2014}$self-attention scheme. We introduce an instance of this class named $\textit{Vision Transformer Cellular Automata}$ (ViTCA). We present quantitative and qualitative results on denoising autoencoding across six benchmark datasets, comparing ViTCA to a U-Net, a U-Net-based CA baseline (UNetCA), and a Vision Transformer (ViT). When comparing across architectures configured to similar parameter complexity, ViTCA architectures yield superior performance across all benchmarks and for nearly every evaluation metric. We present an ablation study on various architectural configurations of ViTCA, an analysis of its effect on cell states, and an investigation on its inductive biases. Finally, we examine its learned representations via linear probes on its converged cell state hidden representations, yielding, on average, superior results when compared to our U-Net, ViT, and UNetCA baselines.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
