arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2106.08704cs.LGcs.PLcs.SE
收藏

神经编码智能模型的泛化与泛化

Memorization and Generalization in Neural Code Intelligence Models

University of Houston(休斯顿大学);Carnegie Mellon University(卡内基梅隆大学)

Md Rafiqul Islam Rabin, Aftab Hussain, Mohammad Amin Alipour, Vincent J. Hellendoorn

摘要:

深度神经网络(DNN)越来越多地用于软件工程和代码智能任务。这些都是强大的工具,能够通过数百万个参数从大型数据集中学习高度概括的模式。同时,它们的大容量可能使它们易于记忆数据点。最近的研究表明,当训练数据集是嘈杂的,涉及许多模糊或可疑的样本时,记忆风险表现得特别强烈,记忆是唯一的手段。本文的目的是评估和比较的程度记忆和泛化的神经代码智能模型。它旨在提供关于记忆如何影响代码智能系统中神经模型的学习行为的见解。为了观察模型中记忆的程度,我们将随机噪声添加到原始训练数据集中,并使用各种指标来量化噪声对训练和测试各个方面的影响。我们评估了几个基于Java、Python和Ruby代码库的最先进的神经代码智能模型和基准测试。我们的研究结果突出了重要的风险:数百万个可训练参数允许神经网络记住任何东西,包括噪声数据,并提供错误的泛化感。我们观察到所有模型都表现出某种形式的记忆。在大多数代码智能任务中,这可能会带来麻烦,因为它们依赖于相当容易产生噪音和重复的数据源,例如来自GitHub的代码。据我们所知,我们提供了第一个研究,以量化记忆效应领域的软件工程和代码智能系统。这项工作提高了人们的认识,并为在代码智能系统中训练神经模型的重要问题提供了新的见解,这些问题通常被软件工程研究人员忽视。

英文摘要:

Deep Neural Networks (DNNs) are increasingly being used in software engineering and code intelligence tasks. These are powerful tools that are capable of learning highly generalizable patterns from large datasets through millions of parameters. At the same time, their large capacity can render them prone to memorizing data points. Recent work suggests that the memorization risk manifests especially strongly when the training dataset is noisy, involving many ambiguous or questionable samples, and memorization is the only recourse. The goal of this paper is to evaluate and compare the extent of memorization and generalization in neural code intelligence models. It aims to provide insights on how memorization may impact the learning behavior of neural models in code intelligence systems. To observe the extent of memorization in models, we add random noise to the original training dataset and use various metrics to quantify the impact of noise on various aspects of training and testing. We evaluate several state-of-the-art neural code intelligence models and benchmarks based on Java, Python, and Ruby codebases. Our results highlight important risks: millions of trainable parameters allow the neural networks to memorize anything, including noisy data, and provide a false sense of generalization. We observed all models manifest some forms of memorization. This can be potentially troublesome in most code intelligence tasks where they rely on rather noise-prone and repetitive data sources, such as code from GitHub. To the best of our knowledge, we provide the first study to quantify memorization effects in the domain of software engineering and code intelligence systems. This work raises awareness and provides new insights into important issues of training neural models in code intelligence systems that are usually overlooked by software engineering researchers.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑