arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2411.01074cs.LG
收藏

通过激活驱动训练实现DNN模块化

DNN Modularization via Activation-Driven Training

University of Southern California(南加州大学)

Tuan Ngo, Abid Hassan, Saad Shafiq, Nenad Medvidovic

AI总结:

MODA通过激活驱动的模块化训练方法,减少训练时间并提升模型模块化效果,同时保持原有准确率。

摘要:

深度神经网络(DNNs)在适应不断变化的需求时往往积累技术债务,并面临显著的重新训练成本。将DNN模块化有助于提高其重用性。先前的工作提出了在训练过程中和训练后分解DNN模型为模块的技术。然而,这些策略存在若干缺点,包括模块间显著的权重重叠和准确率损失,仅关注卷积层,以及通过引入辅助掩码来控制模块化而增加的复杂性和训练时间。在本工作中,我们提出MODA,一种激活驱动的模块化训练方法。MODA通过直接调节DNN各层的激活输出,基于三个模块化目标:类内亲和力、类间分散性和紧凑性,促进DNN模型内部的固有模块化。MODA使用三个著名的DNN模型和五个大小各异的数据集进行评估。该评估表明,与现有最先进方法相比,使用MODA具有若干优势:(1)MODA在训练时间上减少了22%;(2)由MODA生成的模块包含多达24倍更少的权重和37倍更少的权重重叠,同时(3)在不进行额外微调的情况下保持原始模型的准确率;在模块替换场景中,(4)MODA在平均情况下提高了目标类的准确率12%,同时对其他类的准确率影响最小。

英文摘要:

Deep Neural Networks (DNNs) tend to accrue technical debt and suffer from significant retraining costs when adapting to evolving requirements. Modularizing DNNs offers the promise of improving their reusability. Previous work has proposed techniques to decompose DNN models into modules both during and after training. However, these strategies yield several shortcomings, including significant weight overlaps and accuracy losses across modules, restricted focus on convolutional layers only, and added complexity and training time by introducing auxiliary masks to control modularity. In this work, we propose MODA, an activation-driven modular training approach. MODA promotes inherent modularity within a DNN model by directly regulating the activation outputs of its layers based on three modular objectives: intra-class affinity, inter-class dispersion, and compactness. MODA is evaluated using three well-known DNN models and five datasets with varying sizes. This evaluation indicates that, compared to the existing state-of-the-art, using MODA yields several advantages: (1) MODA accomplishes modularization with 22% less training time; (2) the resultant modules generated by MODA comprise up to 24x fewer weights and 37x less weight overlap while (3) preserving the original model's accuracy without additional fine-tuning; in module replacement scenarios, (4) MODA improves the accuracy of a target class by 12% on average while ensuring minimal impact on the accuracy of other classes.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑