MPC友好的Softmax替代品的有效性
Effectiveness of MPC-friendly Softmax Replacement
摘要：
Softmax广泛用于深度学习，将某些表示映射到概率分布。由于它基于exp/log函数，在多方计算中相对昂贵，Mohassel和Zhang（2017）提出了一种基于ReLU的更简单的替代方案，用于安全计算。然而，我们无法重现他们在MNIST上使用三个完全连接层进行训练时报告的准确性。后来的作品（例如，Wagh等人，2019年和2021年）使用softmax替代不是为了计算输出概率分布，而是为了近似反向传播中的梯度。在这项工作中，我们分析了替换的两种用途，并将其与softmax进行了比较，无论是在多方计算的准确性还是成本方面。我们发现，这种替换只为单层网络提供了显著的速度提升，但它总是降低准确性，有时会显着降低。因此，我们得出结论，它的用处是有限的，应该使用原始的softmax函数。
英文摘要：
Softmax is widely used in deep learning to map some representation to a probability distribution. As it is based on exp/log functions that are relatively expensive in multi-party computation, Mohassel and Zhang (2017) proposed a simpler replacement based on ReLU to be used in secure computation. However, we could not reproduce the accuracy they reported for training on MNIST with three fully connected layers. Later works (e.g., Wagh et al., 2019 and 2021) used the softmax replacement not for computing the output probability distribution but for approximating the gradient in back-propagation. In this work, we analyze the two uses of the replacement and compare them to softmax, both in terms of accuracy and cost in multi-party computation. We found that the replacement only provides a significant speed-up for a one-layer network while it always reduces accuracy, sometimes significantly. Thus we conclude that its usefulness is limited and one should use the original softmax function instead.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
