arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2306.09376cs.LGcs.AIcs.SE
收藏

训练同时模块化:一种DNN模型模块化的新范式

Modularizing while Training: A New Paradigm for Modularizing DNN Models

Beihang University(北京航空航天大学);Chongqing University(重庆大学)

Binhang Qi, Hailong Sun, Hongyu Zhang, Ruobing Zhao, Xiang Gao

AI总结:

针对现有训练后DNN模块化方法开销大、精度损失高的问题,提出训练同时模块化范式MwT,通过优化内聚与耦合的损失函数训练模块化CNN模型,在精度、模块精简度和时间成本上均优于现有最优方法。

摘要:

深度神经网络(DNN)模型已成为智能软件系统中日益关键的组成部分。然而,训练一个DNN模型通常在时间和成本上都十分昂贵。为解决这一问题,研究人员近期借鉴软件工程中代码复用的思路,聚焦于复用现有DNN模型。但复用整个模型可能会带来额外开销,或继承非预期功能的缺陷。因此,现有研究提出将已训练好的模型分解为模块,即“训练后模块化”,以实现模块复用。由于已训练模型并非为模块化设计,训练后模块化会产生巨大开销并造成模型精度损失。本文提出一种将模块化融入模型训练过程的新方法,即“训练同时模块化(MwT)”。我们通过两个分别优化模块内聚性和模块间耦合度的损失函数,训练出结构上具备模块化特性的模型。本研究已实现了针对卷积神经网络(CNN)模型的所提模块化方法。在代表性模型上的评估结果表明,MwT的性能优于当前最优方法。具体而言,MwT造成的精度损失仅为1.13个百分点,比基线方法低1.76个百分点;MwT生成模块的核保留率仅为14.58%,较当前最优方法降低了74.31%;此外,训练与模块化的总时间成本仅为108分钟,是基线方法的一半。

英文摘要:

Deep neural network (DNN) models have become increasingly crucial components in intelligent software systems. However, training a DNN model is typically expensive in terms of both time and money. To address this issue, researchers have recently focused on reusing existing DNN models - borrowing the idea of code reuse in software engineering. However, reusing an entire model could cause extra overhead or inherits the weakness from the undesired functionalities. Hence, existing work proposes to decompose an already trained model into modules, i.e., modularizing-after-training, and enable module reuse. Since trained models are not built for modularization, modularizing-after-training incurs huge overhead and model accuracy loss. In this paper, we propose a novel approach that incorporates modularization into the model training process, i.e., modularizing-while-training (MwT). We train a model to be structurally modular through two loss functions that optimize intra-module cohesion and inter-module coupling. We have implemented the proposed approach for modularizing Convolutional Neural Network (CNN) models in this work. The evaluation results on representative models demonstrate that MwT outperforms the state-of-the-art approach. Specifically, the accuracy loss caused by MwT is only 1.13 percentage points, which is 1.76 percentage points less than that of the baseline. The kernel retention rate of the modules generated by MwT is only 14.58%, with a reduction of 74.31% over the state-of-the-art approach. Furthermore, the total time cost required for training and modularizing is only 108 minutes, half of the baseline.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑