arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.23299math.OC
收藏

用于训练ReLU神经网络的高效确定性全局优化的剪枝

Pruning for efficient deterministic global optimization over trained ReLU neural networks

Process Intelligence Research(过程智能研究)

Giacomo Lastrucci, Tanuj Karia, Victor Schulte, Dominik Bongartz, Artur M. Schweidtmann

AI总结:

本文研究了剪枝技术在嵌入神经网络的优化问题中的应用,通过理论分析和实验验证,发现剪枝可显著提升计算效率,尤其在深度网络和工程应用中效果显著。

摘要:

神经网络越来越多地被用作优化问题中的替代模型,以替代计算成本高的模型。然而,将ReLU神经网络嵌入数学规划中会引入显著的计算挑战,特别是对于深度和宽广的网络,由于ReLU断言的公式化以及由此产生的大规模优化问题。本文研究了剪枝技术如何加速解决嵌入神经网络的优化问题,重点探讨了计算增益的机制。我们提供了关于无结构(权重)和结构化(节点)剪枝如何影响ReLU大-M公式化的理论见解,证明剪枝单调地收紧了预激活界限。我们通过多个网络架构的综合实证研究,使用示例测试函数和现实的化工流程优化案例研究。结果表明,剪枝可实现高达三到四个数量级的速度提升,计算增益归因于三个关键因素:(i)问题规模的减少,(ii)整数变量数量的减少,(iii)大-M界限的收紧。权重剪枝对深度、狭窄的网络特别有效,而节点剪枝在浅层、宽或中等大小的网络中表现更好。在化工工程案例研究中,剪枝使原本不可解的问题在几秒钟内收敛。我们建议在开发神经网络替代模型进行优化时采用剪枝作为标准实践,特别是在需要重复优化求解的工程应用中。

英文摘要:

Neural networks are increasingly used as surrogates in optimization problems to replace computationally expensive models. However, embedding ReLU neural networks in mathematical programs introduces significant computational challenges, particularly for large networks. This work investigates how pruning techniques can accelerate the solution of optimization problems with embedded neural networks, focusing on the mechanisms underlying the computational gains. We show that, besides problem size reduction, both unstructured and structured pruning tightens preactivation bounds. We conduct a comprehensive empirical study spanning three morphologies and four complexity levels, comparing magnitude-based weight and node pruning with structured perspective regularization (SPR). We test on both an illustrative test function and a chemical process flowsheet surrogate, using a commercial and an open-source solver. Pruning consistently improves the accuracy-runtime trade-off, with speedups spanning one to four orders of magnitude depending on the complexity of the unpruned network. The gains are attributed to three factors: (i) reduction in problem size, (ii) decrease in the number of integer variables, and (iii) tightening of big-M bounds. Weight pruning and SPR occupy most of the accuracy-runtime Pareto frontier, the latter enabling faster convergence at the mildest sparsity considered. The mechanisms transfer from the test function to the engineering case study, although the preferred technique and sparsity level might vary with the case study, and should be selected as part of surrogate development.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑