通过学习的结构丢弃和输入依赖门控实现动态门MLP条件计算
DynamicGate MLP Conditional Computation via Learned Structural Dropout and Input Dependent Gating for Functional Plasticity
AI总结：
本文提出动态门MLP框架，结合结构丢弃和输入依赖门控，实现条件计算与正则化双重目标，提升计算效率。
摘要：
Dropout是一种代表性正则化技术，通过在训练中随机失活隐藏单元来缓解过拟合。相比之下，标准推断执行完整网络进行密集计算，其目标和机制与条件计算不同，后者执行操作依赖于输入。本文将DynamicGate-MLP组织为单一框架，同时满足正则化视图和条件计算视图。代替随机掩码，所提模型学习门控决定是否使用每个单元（或块），在实现样本依赖执行时抑制不必要的计算，将计算集中在每个输入所需部分。为此，我们定义连续门控概率，并在推断时从其中生成离散执行掩码以选择执行路径。训练通过惩罚预期门控使用来控制计算预算，并使用Straight-Through Estimator (STE) 优化离散掩码。我们在MNIST、CIFAR-10、Tiny-ImageNet、Speech Commands和PBMC3k上评估DynamicGate-MLP，并与各种MLP基线和MoE风格变体进行比较。计算效率在一致标准下通过门控激活比率和层加权相对MAC指标进行比较，而不是依赖硬件和后端内核的墙钟延迟。
英文摘要：
Dropout is a representative regularization technique that stochastically deactivates hidden units during training to mitigate overfitting. In contrast, standard inference executes the full network with dense computation, so its goal and mechanism differ from conditional computation, where the executed operations depend on the input. This paper organizes DynamicGate-MLP into a single framework that simultaneously satisfies both the regularization view and the conditional-computation view. Instead of a random mask, the proposed model learns gates that decide whether to use each unit (or block), suppressing unnecessary computation while implementing sample-dependent execution that concentrates computation on the parts needed for each input. To this end, we define continuous gate probabilities and, at inference time, generate a discrete execution mask from them to select an execution path. Training controls the compute budget via a penalty on expected gate usage and uses a Straight-Through Estimator (STE) to optimize the discrete mask. We evaluate DynamicGate-MLP on MNIST, CIFAR-10, Tiny-ImageNet, Speech Commands, and PBMC3k, and compare it with various MLP baselines and MoE-style variants. Compute efficiency is compared under a consistent criterion using gate activation ratios and a layerweighted relative MAC metric, rather than wall-clock latency that depends on hardware and backend kernels.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
