在本文中，我们提出了一种新的隐式数据表示的方法，使用一个训练好的神经网络来评估输入数据的相似性。与之前使用梯度表示的方法相反，我们只利用神经网络最后一个隐藏层的输出，而不使用向后的步骤。所提出的技术明确地考虑到初始任务，并显着减少了向量表示的大小，以及计算时间。关键点是最小化层间的信息损失。通常，神经网络丢弃与问题无关的信息，这使得最后的隐藏层表示对于输入相似性任务无用。在这项工作中，我们考虑两个主要原因的信息丢失：神经元之间的相关性和最后隐藏层的大小不足。为了减少神经元之间的相关性，我们对每一层使用正交权重初始化，并修改损失函数以确保训练期间权重的正交性。此外，我们表明，激活函数可以潜在地增加相关性。为了解决这个问题，我们应用了修改的Batch-Normalization with Dropout。使用正交权重矩阵允许我们将这种神经网络视为随机投影方法的应用，并得到最后一个隐藏层大小的下界估计。我们在MNIST和体检数据集上进行实验。在这两个实验中，首先，我们将一组标签分成两个不相交的子集，以训练一个用于二进制分类问题的神经网络，然后使用该模型来测量输入数据之间的相似性并定义隐藏类。我们的实验结果表明，该方法在输入相似性任务上取得了有竞争力的结果，同时减少了计算时间和输入表示的大小。
英文摘要：
In this paper, we propose a novel approach for implicit data representation to evaluate similarity of input data using a trained neural network. In contrast to the previous approach, which uses gradients for representation, we utilize only the outputs from the last hidden layer of a neural network and do not use a backward step. The proposed technique explicitly takes into account the initial task and significantly reduces the size of the vector representation, as well as the computation time. The key point is minimization of information loss between layers. Generally, a neural network discards information that is not related to the problem, which makes the last hidden layer representation useless for input similarity task. In this work, we consider two main causes of information loss: correlation between neurons and insufficient size of the last hidden layer. To reduce the correlation between neurons we use orthogonal weight initialization for each layer and modify the loss function to ensure orthogonality of the weights during training. Moreover, we show that activation functions can potentially increase correlation. To solve this problem, we apply modified Batch-Normalization with Dropout. Using orthogonal weight matrices allow us to consider such neural networks as an application of the Random Projection method and get a lower bound estimate for the size of the last hidden layer. We perform experiments on MNIST and physical examination datasets. In both experiments, initially, we split a set of labels into two disjoint subsets to train a neural network for binary classification problem, and then use this model to measure similarity between input data and define hidden classes. Our experimental results show that the proposed approach achieves competitive results on the input similarity task while reducing both computation time and the size of the input representation.
Fraunhofer IAIS（弗劳恩霍夫智能分析与信息系统研究所）；Competence Center Machine Learning Rhine-Ruhr（莱茵-鲁尔机器学习能力中心）；University of Bonn（波恩大学）；University of Nottingham（诺丁汉大学）
Enhancing Pretrained Model-based Continual Representation Learning via Guided Random Projection
Wuhan University（武汉大学）；Shanghai Innovation Institute（上海创新研究院）；Shanghai Jiao Tong University（上海交通大学）；Tsinghua University（清华大学）；University of Science and Technology of China（中国科学技术大学）；China University of Mining and Technology（中国矿业大学）
Feature Map Similarity Reduction in Convolutional Neural Networks
Cranfield University（克兰菲尔德大学）；Centre for Computational Engineering Sciences（计算工程科学中心）；Inria（法国国家信息与自动化技术研究院）；Univ. Bordeaux（波尔多大学）；ESTIA-Institute of Technology（ESTIA技术研究所）
