SILO语言模型：在非参数数据存储中隔离法律风险
SILO Language Models: Isolating Legal Risk In a Nonparametric Datastore
AI总结：
SILO通过将参数化模型训练于228B token的开放许可语料库，并在推理时查询可修改的非参数数据存储，在不训练高风险数据的前提下缩小90%性能差距，实现法律风险与性能的平衡。
摘要：
在受版权保护或其他受限数据上训练语言模型（LMs）的合法性正面临激烈争论。然而，正如我们所展示的，如果仅在低风险文本（例如，已进入公有领域的书籍或政府文件）上训练，模型性能会因其规模有限和领域覆盖不足而显著下降。我们提出了SILO，这是一种在推理阶段管理风险与性能权衡的新型语言模型。SILO的构建方式为：（1）在开放许可语料库（OLC）上训练一个参数化语言模型，OLC是我们新整理的语料库，包含228B个token的公有领域和宽松许可文本；（2）为其增加一个更通用且易于修改的非参数数据存储（例如，包含受版权保护的书籍或新闻），该数据存储仅在推理阶段被查询。该数据存储允许在不进行训练的情况下使用高风险数据，支持句子级数据溯源，并使数据生产者能够通过从存储中移除内容来选择退出该模型。这些能力有助于促进对数据使用法规的合规性，例如美国的合理使用原则和欧盟的GDPR。我们的实验表明，参数化语言模型在OLC未覆盖的领域上表现困难。然而，访问数据存储极大地改善了领域外性能，缩小了与在Pile上训练的语言模型之间90%的性能差距，Pile是一个更多样化但大多包含高风险文本的语料库。我们还分析了哪种非参数方法效果最好、剩余错误位于何处，以及性能如何随数据存储规模变化。我们的结果表明，在降低法律风险的同时构建高质量语言模型是可能的。
英文摘要：
The legality of training language models (LMs) on copyrighted or otherwise restricted data is under intense debate. However, as we show, model performance significantly degrades if trained only on low-risk text (e.g., out-of-copyright books or government documents), due to its limited size and domain coverage. We present SILO, a new language model that manages this risk-performance tradeoff during inference. SILO is built by (1) training a parametric LM on Open License Corpus (OLC), a new corpus we curate with 228B tokens of public domain and permissively licensed text and (2) augmenting it with a more general and easily modifiable nonparametric datastore (e.g., containing copyrighted books or news) that is only queried during inference. The datastore allows use of high-risk data without training on it, supports sentence-level data attribution, and enables data producers to opt out from the model by removing content from the store. These capabilities can foster compliance with data-use regulations such as the fair use doctrine in the United States and the GDPR in the European Union. Our experiments show that the parametric LM struggles on domains not covered by OLC. However, access to the datastore greatly improves out of domain performance, closing 90% of the performance gap with an LM trained on the Pile, a more diverse corpus with mostly high-risk text. We also analyze which nonparametric approach works best, where the remaining errors lie, and how performance scales with datastore size. Our results suggest that it is possible to build high quality language models while mitigating their legal risk.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
