A new method for the unsupervised learning of sparse representations using autoencoders is proposed and implemented by ordering the output of the hidden units by their activation value and progressively reconstructing the input in this order. This can be done efficiently in parallel with the use of cumulative sums and sorting only slightly increasing the computational costs. Minimizing the difference of this progressive reconstruction with respect to the input can be seen as minimizing the number of active output units required for the reconstruction of the input. The model thus learns to reconstruct optimally using the least number of active output units. This leads to high sparsity without the need for extra hyperparameters, the amount of sparsity is instead implicitly learned by minimizing this progressive reconstruction error. Results of the trained model are given for patches of the CIFAR10 dataset, showing rapid convergence of features and extremely sparse output activations while maintaining a minimal reconstruction error and showing extreme robustness to overfitting. Additionally the reconstruction as function of number of active units is presented which shows the autoencoder learns a rank order code over the input where the highest ranked units correspond to the highest decrease in reconstruction error.
虽然已经有很多工作致力于理解监督环境中深度非线性网络的隐式(和显式)正则化,但本文关注的是无监督学习,即,训练自动编码器的目的是从输入再现输出。我们扩展了最近关于无约束线性模型的结果[Jin et al. 2021],并将其应用于(1)非线性自编码器和(2)约束线性自编码器,得到以下两个结果:首先,我们证明了无监督设置本身会引起强的额外正则化,即,学习的自动编码器的模型容量的严重降低:我们得出,如果两个模型在它们的最后隐藏层中具有相同的维数(并且在一些附加假设下),则深度非线性自动编码器不能比线性自动编码器更准确地拟合训练数据。我们的第二个贡献是关于低秩EDLAE模型[Steck 2020],这是一个线性自动编码器,对学习的低秩参数矩阵的对角线有约束,以提高泛化能力:我们推导出其非凸训练目标的最佳值的封闭形式近似,并实证证明,它是一个准确的近似在所有的模型排名在我们的实验上的三个众所周知的数据集。
Variational Rank Reduction Autoencoders for Generative Thermal Design
ESI Group-UZ Chair of the National Strategy on Artificial Intelligence(ESI集团-乌兹主席国家人工智能战略办公室);Aragon Institute of Engineering Research (I3A)(阿拉贡工程研究所(I3A));Universidad de Zaragoza(萨拉戈萨大学);PIMM Lab(PIMM实验室);ENSAM Institute of Technology(ENSAM技术研究所)