高效稀疏训练与结构化Dropout
Efficient Sparse Training with Structured Dropout
AI总结：
提出结构化Dropout变体SparseDrop，利用硬件友好的稀疏性实现GPU加速，在保持或提升正则化效果的同时，作为标准Dropout的替代品加速训练。
摘要：
Dropout是深度学习中一种常见的正则化技术，用于改善泛化能力。尽管它引入了稀疏性，从而具有提高吞吐量的潜力，但由于其非结构化特性，通常无法在GPU上带来加速。在本项目中，我实验了SparseDrop，一种结构化的、硬件友好的Dropout变体，能够利用这种稀疏性。我提供了SparseDrop的CUDA实现，即使在低稀疏度水平下也能相对于其密集对应物实现加速。实证结果表明，SparseDrop提供了与标准Dropout相似甚至有时更好的正则化特性。这表明它有可能作为标准Dropout的直接替代品，实现更快的训练速度。源代码可在https://github.com/andylolu2/sparse-dropout获取。
英文摘要：
Dropout is a common regularisation technique in deep learning that improves generalisation. Even though it introduces sparsity and thus potential for higher throughput, it usually cannot bring speed-ups on GPUs due to its unstructured nature. In this project, I experiment with SparseDrop, a structured, hardware-friendly variant of dropout that can exploit such sparsity. I provide a CUDA implementation of SparseDrop, achieving speed-ups against its dense counterpart even at low sparsity levels. The empirical results demonstrate that SparseDrop provides similar, or sometimes even better, regularisation properties as standard dropout. This suggests its potential as a drop-in replacement to standard dropout with faster training speeds. The source code is available at https://github.com/andylolu2/sparse-dropout
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
