TUTOR:使用决策规则作为模型先验训练神经网络
TUTOR: Training Neural Networks Using Decision Rules as Model Priors
摘要:
人类的大脑有能力在有限的经验下完成新的任务。它利用以前的学习经验,使解决方案策略适应新的领域。另一方面,深度神经网络(DNN)通常需要大量数据和计算资源用于训练。然而,这一要求在许多情况下并没有得到满足。为了应对这些挑战,我们提出了TUTOR DNN合成框架。TUTOR针对表格数据集。它综合了精确的DNN模型,具有有限的可用数据和减少的内存/计算需求。它由三个连续的步骤组成。第一步涉及合成数据的生成、验证和标记。合成数据生成模块以分类特征和连续特征两者为目标。TUTOR根据与真实数据相同的概率分布生成合成数据。然后使用语义完整性分类器模块验证所生成的合成数据的完整性。它基于从真实数据集中提取的一组规则来标记合成数据。接下来,TUTOR使用两种训练方案,将合成数据和训练数据相结合,以学习DNN模型的参数。这两种方案侧重于两种不同的方式,其中合成数据可用于导出模型参数的先验,因此为使用真实数据进行训练提供更好的DNN初始化。在第三步中,TUTOR采用生长和修剪合成范式来学习DNN的权重和架构,以减少模型大小,同时确保其准确性。我们评估了九个不同大小的数据集上的TUTOR性能。我们发现,与完全连接的DNN相比,TUTOR平均减少了5.9倍的数据需求,提高了3.4%的准确性,并减少了4.7倍(4.3倍)的参数数量(fFLOP)。因此,TUTOR可以实现更少的数据需求,更准确,更紧凑的DNN合成。
英文摘要:
The human brain has the ability to carry out new tasks with limited experience. It utilizes prior learning experiences to adapt the solution strategy to new domains. On the other hand, deep neural networks (DNNs) generally need large amounts of data and computational resources for training. However, this requirement is not met in many settings. To address these challenges, we propose the TUTOR DNN synthesis framework. TUTOR targets tabular datasets. It synthesizes accurate DNN models with limited available data and reduced memory/computational requirements. It consists of three sequential steps. The first step involves generation, verification, and labeling of synthetic data. The synthetic data generation module targets both the categorical and continuous features. TUTOR generates the synthetic data from the same probability distribution as the real data. It then verifies the integrity of the generated synthetic data using a semantic integrity classifier module. It labels the synthetic data based on a set of rules extracted from the real dataset. Next, TUTOR uses two training schemes that combine synthetic and training data to learn the parameters of the DNN model. These two schemes focus on two different ways in which synthetic data can be used to derive a prior on the model parameters and, hence, provide a better DNN initialization for training with real data. In the third step, TUTOR employs a grow-and-prune synthesis paradigm to learn both the weights and the architecture of the DNN to reduce model size while ensuring its accuracy. We evaluate the performance of TUTOR on nine datasets of various sizes. We show that in comparison to fully connected DNNs, TUTOR, on an average, reduces the need for data by 5.9x, improves accuracy by 3.4%, and reduces the number of parameters (fFLOPs) by 4.7x (4.3x). Thus, TUTOR enables a less data-hungry, more accurate, and more compact DNN synthesis.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。