大型语言模型训练数据集中代码许可证侵权的探索性研究
An Exploratory Investigation into Code License Infringements in Large Language Model Training Datasets
AI总结：
本研究调查了大型语言模型训练数据集中的代码许可证侵权问题，发现53个模型的数据集中普遍存在许可证不一致现象，建议社区优先制定数据集管理的最佳实践。
摘要：
大型语言模型的训练是否会潜在侵犯代码许可证？此外，是否存在可以在不违反此类许可证的情况下安全用于训练这些模型的数据集？在我们的研究中，我们评估了该领域的当前趋势以及将代码纳入大型语言模型训练的重要性。此外，我们检查了公开可用的数据集，以查看是否可以在其上训练这些模型而不会在未来面临法律问题的风险。为此，我们编制了一份包含53个在文件级代码上训练的大型语言模型的列表。然后，我们提取了它们的数据集，并分析了它们与我们创建的一个仅包含强传染性开源许可证代码的数据集的重叠程度。我们的分析显示，尽管我们是根据其关联的代码库许可证进行选择的，但我们检查的每个数据集都包含许可证不一致的情况。我们总共分析了5.14亿个代码文件，发现其中3800万个与我们的强传染性开源许可证数据集中的文件完全重复。此外，我们检查了1.71亿个文件开头注释，识别出1600万个带有强传染性开源许可证的注释，以及另外1100万个不鼓励复制但未明确提及许可证的注释。基于我们的研究结果，这突显了在基于代码训练的大型语言模型中许可证不一致问题的普遍性，我们建议研究人员和社区优先考虑制定和采用数据集创建与管理的最佳实践。
英文摘要：
Does the training of large language models potentially infringe upon code licenses? Furthermore, are there any datasets available that can be safely used for training these models without violating such licenses? In our study, we assess the current trends in the field and the importance of incorporating code into the training of large language models. Additionally, we examine publicly available datasets to see whether these models can be trained on them without the risk of legal issues in the future. To accomplish this, we compiled a list of 53 large language models trained on file-level code. We then extracted their datasets and analyzed how much they overlap with a dataset we created, consisting exclusively of strong copyleft code. Our analysis revealed that every dataset we examined contained license inconsistencies, despite being selected based on their associated repository licenses. We analyzed a total of 514 million code files, discovering 38 million exact duplicates present in our strong copyleft dataset. Additionally, we examined 171 million file-leading comments, identifying 16 million with strong copyleft licenses and another 11 million comments that discouraged copying without explicitly mentioning a license. Based on the findings of our study, which highlights the pervasive issue of license inconsistencies in large language models trained on code, our recommendation for both researchers and the community is to prioritize the development and adoption of best practices for dataset creation and management.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
