Circa：用于私人深度学习的随机ReLUs
Circa: Stochastic ReLUs for Private Deep Learning
摘要：
机器学习作为一种服务的同时兴起以及对用户隐私的担忧越来越多地激发了对私有推理（PI）的需求。虽然最近的工作表明PI是可能的使用密码原语，计算开销使其不切实际。社区在很大程度上还没有准备好解决这些开销，因为PI中的减速来源于ReLU运算符，而对明文推理的优化则集中在优化FLOP上。在本文中，我们重新思考了ReLU计算，并提出了针对神经网络属性的PI优化。具体来说，我们将ReLU重新定义为近似符号测试，并为符号测试引入了一种新的截断方法，可以显着降低每个ReLU的成本。这些优化导致特定类型的随机ReLU。关键的观察是，随机故障行为非常适合神经网络推理的容错特性。因此，我们在不影响精度的情况下提供了显著的节省。我们将这些优化统称为Circa，并展示了与基线实现相比高达4.7倍的存储和3倍的运行时间的改进;我们还展示了Circa可以在最近的PI优化之上使用，以获得1.8倍的额外加速。
英文摘要：
The simultaneous rise of machine learning as a service and concerns over user privacy have increasingly motivated the need for private inference (PI). While recent work demonstrates PI is possible using cryptographic primitives, the computational overheads render it impractical. The community is largely unprepared to address these overheads, as the source of slowdown in PI stems from the ReLU operator whereas optimizations for plaintext inference focus on optimizing FLOPs. In this paper we re-think the ReLU computation and propose optimizations for PI tailored to properties of neural networks. Specifically, we reformulate ReLU as an approximate sign test and introduce a novel truncation method for the sign test that significantly reduces the cost per ReLU. These optimizations result in a specific type of stochastic ReLU. The key observation is that the stochastic fault behavior is well suited for the fault-tolerant properties of neural network inference. Thus, we provide significant savings without impacting accuracy. We collectively call the optimizations Circa and demonstrate improvements of up to 4.7x storage and 3x runtime over baseline implementations; we further show that Circa can be used on top of recent PI optimizations to obtain 1.8x additional speedup.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
