arXiv 2212.059702022-12cs.SEcs.CLcs.LG

将循环神经网络分解为模块以实现可重用性和替换

Decomposing a Recurrent Neural Network into Modules for Enabling Reusability and Replacement

Iowa State University（爱荷华州立大学）；IBM Research（IBM研究院）

Sayem Mohammad Imtiaz, Fraol Batole, Astha Singh, Rangeet Pan, Breno Dantas Cruz, Hridesh Rajan

备注：: Accepted at 45th international conference on software engineering (ICSE'2023)

摘要：

我们是否可以使用经过训练的递归神经网络（RNN）来进行语言之间的翻译，并将其扩展到支持新的自然语言，而无需从头开始重新训练模型？我们可以通过替换与错误行为相关的部分来修复RNN的错误行为吗？最近关于将全连接神经网络（FCNN）和卷积神经网络（CNN）分解为模块的工作已经显示了以这种方式设计深度模型的价值，这在传统SE中是标准的，但对于深度学习模型来说是陌生的。然而，先前的工作集中在基于图像的多类分类问题上，并且由于（a）不同的层结构，（b）循环结构，（c）不同类型的输入-输出架构，以及（d）使用非线性和逻辑激活函数而不能应用于RNN。在这项工作中，我们提出了第一种将RNN分解为模块的方法。我们研究不同类型的RNN，即，Vanilla、LSTM和GRU。此外，我们还展示了如何在各种场景中重用和替换这些RNN模块。我们对5个规范数据集（即，Math QA、Brown Corpus、Wiki-toxicity、Clinc OOS和Tatoeba）和每个数据集的4个模型变量。我们发现，分解一个训练好的模型的成本很小（准确率：-0.6%，BLEU得分：+0.10%）。此外，分解的模块可以重用和替换，而无需重新训练。

英文摘要：

Can we take a recurrent neural network (RNN) trained to translate between languages and augment it to support a new natural language without retraining the model from scratch? Can we fix the faulty behavior of the RNN by replacing portions associated with the faulty behavior? Recent works on decomposing a fully connected neural network (FCNN) and convolutional neural network (CNN) into modules have shown the value of engineering deep models in this manner, which is standard in traditional SE but foreign for deep learning models. However, prior works focus on the image-based multiclass classification problems and cannot be applied to RNN due to (a) different layer structures, (b) loop structures, (c) different types of input-output architectures, and (d) usage of both nonlinear and logistic activation functions. In this work, we propose the first approach to decompose an RNN into modules. We study different types of RNNs, i.e., Vanilla, LSTM, and GRU. Further, we show how such RNN modules can be reused and replaced in various scenarios. We evaluate our approach against 5 canonical datasets (i.e., Math QA, Brown Corpus, Wiki-toxicity, Clinc OOS, and Tatoeba) and 4 model variants for each dataset. We found that decomposing a trained model has a small cost (Accuracy: -0.6%, BLEU score: +0.10%). Also, the decomposed modules can be reused and replaced without needing to retrain.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。