arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2604.04736cs.LGcs.AIcs.DC
收藏

快速高效的贝叶斯学习中的采样并行

Sampling Parallelism for Fast and Efficient Bayesian Learning

Karlsruhe Institute of Technology(卡尔斯鲁厄理工学院);Helmholtz AI(亥姆霍兹人工智能)

Asena Karolin Özdemir, Lars H. Heyen, Arvid Weyrauch, Achim Streit, Markus Götz, Charlotte Debus

AI总结:

本文提出采样并行策略,通过分布式GPU加速采样过程,降低内存压力和训练时间,提升贝叶斯学习效率,同时与数据并行策略结合增强模型鲁棒性。

摘要:

机器学习模型,尤其是深度神经网络,在风险敏感领域如医疗、环境预测和金融中得到越来越多的应用,其中预测不确定性可靠量化至关重要。然而,许多不确定性量化(UQ)方法由于计算成本高而难以应用。基于采样的贝叶斯学习方法,如贝叶斯神经网络(BNNs),尤其昂贵,因为绘制和评估多个参数样本会迅速耗尽内存和计算资源。这些限制已限制了贝叶斯技术的可用性和探索。为了解决这些挑战,我们引入了采样并行,一种简单而强大的并行策略,针对基于采样的贝叶斯学习的主要瓶颈:样本本身。通过在多个GPU上分布样本评估,我们的方法减少内存压力和训练时间,无需架构更改或广泛的超参数调整。我们详细描述了方法,并在几个示例任务和架构上评估其性能,与分布式数据并行(DDP)作为基线进行比较。我们进一步证明,采样并行与现有策略互补,通过实现结合样本和数据并行的混合方法。我们的实验显示,当样本数量与计算资源成比例扩展时,接近完美的扩展性,证实了样本评估可以干净地并行化。尽管DDP在固定工作负载下实现更好的原始加速,但采样并行有显著优势:通过在每个GPU上对同一批次应用独立的随机增强,它增加了增强的多样性,从而减少收敛所需的epoch数量。

英文摘要:

Machine learning models, and deep neural networks in particular, are increasingly deployed in risk-sensitive domains such as healthcare, environmental forecasting, and finance, where reliable quantification of predictive uncertainty is essential. However, many uncertainty quantification (UQ) methods remain difficult to apply due to their substantial computational cost. Sampling-based Bayesian learning approaches, such as Bayesian neural networks (BNNs), are particularly expensive since drawing and evaluating multiple parameter samples rapidly exhausts memory and compute resources. These constraints have limited the accessibility and exploration of Bayesian techniques thus far. To address these challenges, we introduce sampling parallelism, a simple yet powerful parallelization strategy that targets the primary bottleneck of sampling-based Bayesian learning: the samples themselves. By distributing sample evaluations across multiple GPUs, our method reduces memory pressure and training time without requiring architectural changes or extensive hyperparameter tuning. We detail the methodology and evaluate its performance on a few example tasks and architectures, comparing against distributed data parallelism (DDP) as a baseline. We further demonstrate that sampling parallelism is complementary to existing strategies by implementing a hybrid approach that combines sample and data parallelism. Our experiments show near-perfect scaling when the sample number is scaled proportionally to the computational resources, confirming that sample evaluations parallelize cleanly. Although DDP achieves better raw speedups under scaling with constant workload, sampling parallelism has a notable advantage: by applying independent stochastic augmentations to the same batch on each GPU, it increases augmentation diversity and thus reduces the number of epochs required for convergence.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑