ReLU网络中基于优化的界收紧的计算权衡
Computational Tradeoffs of Optimization-Based Bound Tightening in ReLU Networks
AI总结:
本研究针对ReLU网络MILP模型依赖的激活界,探究界的紧致性与模型求解计算量的权衡,并基于网络结构、正则化和舍入的影响给出模型实现指南。
摘要:
过去十年间,采用混合整数线性规划(MILP)模型来表示带有修正线性单元(ReLU)激活函数的神经网络已变得日益普遍。这使得人们能够利用MILP技术来测试(或施压测试)网络的行为,以对抗性方式改进其训练,并将其嵌入优化模型中以利用其预测能力。这类MILP模型大多依赖激活界,即每个神经元输入值的上下界。在本研究中,我们探究了这些界的紧致性与求解所得MILP模型的计算量之间的权衡关系。我们基于网络结构、正则化和舍入的影响,为实现这些模型提供了指导方针。
英文摘要:
The use of Mixed-Integer Linear Programming (MILP) models to represent neural networks with Rectified Linear Unit (ReLU) activations has become increasingly widespread in the last decade. This has enabled the use of MILP technology to test-or stress-their behavior, to adversarially improve their training, and to embed them in optimization models leveraging their predictive power. Many of these MILP models rely on activation bounds. That is, bounds on the input values of each neuron. In this work, we explore the tradeoff between the tightness of these bounds and the computational effort of solving the resulting MILP models. We provide guidelines for implementing these models based on the impact of network structure, regularization, and rounding.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。