上一篇:Softmax忽略的统计信息可以提高分类器的准确性
Money on the Table: Statistical information ignored by Softmax can improve classifier accuracy
摘要:
Softmax是神经网络(NN)中使用的标准最终层,用于总结训练NN中编码的信息并返回预测。然而,Softmax只利用了训练模型中编码的类特定结构的子集,并忽略了潜在的有价值的信息:在训练期间,模型编码类响应分布的阵列$D$,其中$D_{ij}$是$j^{th}$前Softmax读出神经元对$i^{th}$类的响应的分布。给定一个测试样本,Softmax隐式地只使用这个数组$D$中对应于读出神经元对样本真实类的响应的行。利用更多的这个阵列$D$可以提高分类器的准确性,因为两个竞争类的可能性可以在$D$的其他行中编码。为了探索这种潜在的资源,我们开发了一种混合分类器(Softmax池混合,$SPH$),它使用Softmax对高分样本,但对低分样本使用对数似然方法,池的信息从全阵列$D$。我们将$SPH$应用于在矢量化MNIST数据集上训练的模型,以达到不同的准确度。仅在测试时,$SPH$仅替换训练NN中的最后Softmax层。所有训练都与Softmax相同。由于池化分类器在低得分样本上的性能优于Softmax,因此$SPH$使用完全相同的训练模型将测试集误差降低了6%至23%,而无论Softmax的基线精度如何。这种误差的减少反映了Softmax未使用的训练NN的隐藏容量。
英文摘要:
Softmax is a standard final layer used in Neural Nets (NNs) to summarize information encoded in the trained NN and return a prediction. However, Softmax leverages only a subset of the class-specific structure encoded in the trained model and ignores potentially valuable information: During training, models encode an array $D$ of class response distributions, where $D_{ij}$ is the distribution of the $j^{th}$ pre-Softmax readout neuron's responses to the $i^{th}$ class. Given a test sample, Softmax implicitly uses only the row of this array $D$ that corresponds to the readout neurons' responses to the sample's true class. Leveraging more of this array $D$ can improve classifier accuracy, because the likelihoods of two competing classes can be encoded in other rows of $D$. To explore this potential resource, we develop a hybrid classifier (Softmax-Pooling Hybrid, $SPH$) that uses Softmax on high-scoring samples, but on low-scoring samples uses a log-likelihood method that pools the information from the full array $D$. We apply $SPH$ to models trained on a vectorized MNIST dataset to varying levels of accuracy. $SPH$ replaces only the final Softmax layer in the trained NN, at test time only. All training is the same as for Softmax. Because the pooling classifier performs better than Softmax on low-scoring samples, $SPH$ reduces test set error by 6% to 23%, using the exact same trained model, whatever the baseline Softmax accuracy. This reduction in error reflects hidden capacity of the trained NN that is left unused by Softmax.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。