大型语言模型抄袭:通过知识图谱比对确保大型语言模型训练数据的负责任来源
LLMs Plagiarize: Ensuring Responsible Sourcing of Large Language Model Training Data Through Knowledge Graph Comparison
AI总结:
本文提出一种基于RDF三元组构建知识图谱的抄袭检测变体系统,通过余弦相似度和归一化图编辑距离从内容与结构两方面评估源文档与LLM续写的相似性,无需访问困惑度等内部指标或训练语料库,即可判断LLM是否在训练或微调中使用了受版权保护的语料。
摘要:
鉴于近期出版商、报纸及其他受版权保护语料库的创作者对大型语言模型开发者提起的法律指控——这些开发者在训练或微调过程中使用了他们的受版权保护材料——我们提出一种新颖的系统,作为抄袭检测系统的一种变体,用于评估某一知识来源是否被用于大型语言模型的训练或微调。与现有方法不同,我们采用一种利用资源描述框架(RDF)三元组的方法,从源文档和大型语言模型对该文档的续写中分别构建知识图谱。随后,在内容方面使用余弦相似度对这些图谱进行分析,在结构方面则使用一种归一化的图编辑距离进行分析,该距离能够展示同构程度。与专注于源语料库和目标语料库之间内容匹配及关键词识别的传统抄袭系统不同,我们的方法通过关注思想之间的关系及其相对于其他思想的组织方式,能够对源文档与大型语言模型续写之间的相似性进行更广泛、更准确的评估。此外,我们的方法不需要访问大型语言模型的指标(如困惑度),这些指标在封闭的大型语言模型“黑盒”系统中以及训练语料库中可能无法获得。因此,我们通过相似性度量来评估大型语言模型在其续写中是否“抄袭”了某一语料库。我们系统的原型将在超链接的GitHub代码库中提供。
英文摘要:
In light of recent legal allegations brought by publishers, newspapers, and other creators of copyrighted corpora against large language model developers who use their copyrighted materials for training or fine-tuning purposes, we propose a novel system, a variant of a plagiarism detection system, that assesses whether a knowledge source has been used in the training or fine-tuning of a large language model. Unlike current methods, we utilize an approach that uses Resource Description Framework (RDF) triples to create knowledge graphs from both a source document and an LLM continuation of that document. These graphs are then analyzed with respect to content using cosine similarity and with respect to structure using a normalized version of graph edit distance that shows the degree of isomorphism. Unlike traditional plagiarism systems that focus on content matching and keyword identification between a source and a target corpus, our approach enables a broader and more accurate evaluation of similarity between a source document and LLM continuation by focusing on relationships between ideas and their organization with regards to others. Additionally, our approach does not require access to LLM metrics like perplexity that may be unavailable in closed large language model "black-box" systems, as well as the training corpus. We thus assess whether an LLM has "plagiarized" a corpus in its continuation through similarity measures. A prototype of our system will be found on a hyperlinked GitHub repository.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。