通过混合精度提升科学机器学习的速度并降低内存占用
Speeding up and reducing memory usage for scientific machine learning via mixed precision
AI总结：
针对半精度训练无法应用于科学机器学习的问题，该研究提出结合float16与float32的混合精度方法，在保持模型精度的同时大幅提升训练速度、降低内存占用，并通过理论分析佐证了实验结果。
摘要：
科学机器学习（SciML）已成为解决复杂计算科学与工程问题的通用方法。在该领域中，物理信息神经网络（PINNs）和深度算子网络（DeepONets）作为结合物理方程与实验数据求解偏微分方程的主流技术脱颖而出。然而，训练PINNs和DeepONets需要大量计算资源，包括漫长的计算时间和巨量内存。为提升计算效率，使用半精度（float16）而非传统单精度（float32）或双精度（float64）训练神经网络的方法因具备减少计算时间与内存消耗的固有优势，受到了广泛关注。但我们发现float16无法应用于SciML方法，原因在于训练初期会出现梯度发散、权重更新归零，且无法收敛至局部最小值。为克服这些局限，我们探索了混合精度方法，该方法结合float16与float32数值格式，以降低内存占用并提升计算速度。我们的实验表明，混合精度训练不仅大幅缩短了训练时间、降低了内存需求，还能保持模型精度。我们还通过理论分析佐证了这些实验观察结果。该研究对各类计算应用中的SciML具有广泛意义。
英文摘要：
Scientific machine learning (SciML) has emerged as a versatile approach to address complex computational science and engineering problems. Within this field, physics-informed neural networks (PINNs) and deep operator networks (DeepONets) stand out as the leading techniques for solving partial differential equations by incorporating both physical equations and experimental data. However, training PINNs and DeepONets requires significant computational resources, including long computational times and large amounts of memory. In search of computational efficiency, training neural networks using half precision (float16) rather than the conventional single (float32) or double (float64) precision has gained substantial interest, given the inherent benefits of reduced computational time and memory consumed. However, we find that float16 cannot be applied to SciML methods, because of gradient divergence at the start of training, weight updates going to zero, and the inability to converge to a local minima. To overcome these limitations, we explore mixed precision, which is an approach that combines the float16 and float32 numerical formats to reduce memory usage and increase computational speed. Our experiments showcase that mixed precision training not only substantially decreases training times and memory demands but also maintains model accuracy. We also reinforce our empirical observations with a theoretical analysis. The research has broad implications for SciML in various computational applications.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
