arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2303.12733cs.CVcs.AI
收藏

关于LAION-2B的去重

On the De-duplication of LAION-2B

Unicaen(卡昂大学);ENSICAEN(卡昂国立高等工程师学院)

Ryan Webster, Julien Rabin, Loic Simon, Frederic Jurie

摘要:

生成模型,如DALL-E,中途和稳定扩散,具有超出计算机科学领域的社会影响。这些模型需要像LAION-2B这样的大型图像数据库,其中包含20亿张图像。在这种规模下,人工检测很困难,自动分析也很有挑战性。此外,最近的研究表明,重复的图像会给在LAION 2B上训练的模型带来版权问题,这会阻碍其可用性。本文提出了一种算法链,该算法链以适度的计算运行,压缩CLIP特征以实现高效的重复检测,即使是对于巨大的图像量。我们的方法表明,大约7亿张图像,或约30%,LAION-2B的图像可能是重复的。我们的方法还提供了这个数据集上的重复直方图,我们使用它来揭示更多的稳定扩散逐字复制的例子,并进一步证明这种方法。消除重复数据集的当前版本将在线分发。

英文摘要:

Generative models, such as DALL-E, Midjourney, and Stable Diffusion, have societal implications that extend beyond the field of computer science. These models require large image databases like LAION-2B, which contain two billion images. At this scale, manual inspection is difficult and automated analysis is challenging. In addition, recent studies show that duplicated images pose copyright problems for models trained on LAION2B, which hinders its usability. This paper proposes an algorithmic chain that runs with modest compute, that compresses CLIP features to enable efficient duplicate detection, even for vast image volumes. Our approach demonstrates that roughly 700 million images, or about 30\%, of LAION-2B's images are likely duplicated. Our method also provides the histograms of duplication on this dataset, which we use to reveal more examples of verbatim copies by Stable Diffusion and further justify the approach. The current version of the de-duplicated set will be distributed online.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑