随机梯度下降中多层蒙特卡洛方法的并行复杂度研究
On the Parallel Complexity of Multilevel Monte Carlo in Stochastic Gradient Descent
AI总结：
针对多层蒙特卡洛（MLMC）在大规模并行平台上扩展性差的问题，提出延迟MLMC梯度估计器，通过复用历史梯度分量降低并行复杂度，在深度对冲实验中验证了其并行性能优势。
摘要：
在用于神经随机微分方程等序列模拟的随机梯度下降（SGD）中，多层蒙特卡洛（MLMC）方法的理论计算复杂度优于朴素蒙特卡洛方法。但在实际应用中，MLMC在现代GPU等大规模并行计算平台上扩展性很差，因为其并行复杂度与朴素蒙特卡洛方法相当。为解决这一问题，我们提出延迟MLMC梯度估计器，通过复用SGD前期步骤中已计算的梯度分量，大幅降低MLMC的并行复杂度。所提估计器可证明能降低每次迭代的平均并行复杂度，代价是单次迭代的收敛速率略有下降。在数值实验中，我们以深度对冲为例，证明该方法的并行复杂度优于SGD中的标准MLMC。
英文摘要：
In the stochastic gradient descent (SGD) for sequential simulations such as the neural stochastic differential equations, the Multilevel Monte Carlo (MLMC) method is known to offer better theoretical computational complexity compared to the naive Monte Carlo approach. However, in practice, MLMC scales poorly on massively parallel computing platforms such as modern GPUs, because of its large parallel complexity which is equivalent to that of the naive Monte Carlo method. To cope with this issue, we propose the delayed MLMC gradient estimator that drastically reduces the parallel complexity of MLMC by recycling previously computed gradient components from earlier steps of SGD. The proposed estimator provably reduces the average parallel complexity per iteration at the cost of a slightly worse per-iteration convergence rate. In our numerical experiments, we use an example of deep hedging to demonstrate the superior parallel complexity of our method compared to the standard MLMC in SGD.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
