Koala：一种量化与预训练语料库重叠的指标
Koala: An Index for Quantifying Overlaps with Pre-training Corpora
摘要：
在最近几年中，更多的注意力已经放在探测预训练数据在大语言模型（LLM）下游行为中的作用上。尽管重要性，有没有公共工具，支持这种分析的预培训语料库在大规模。为了帮助这一领域的研究，我们推出了Koala，这是一种针对大型预训练语料库的可搜索索引，使用具有高效压缩率和搜索支持的压缩后缀数组。在首个版本中，我们索引了OPT 175 B预训练数据的公开比例。Koala提供了一个框架，可以对当前和未来的基准进行法医分析，并评估LLM输出的记忆程度。Koala可在https://koala-index.erc.monash.edu/上供公众使用。
英文摘要：
In very recent years more attention has been placed on probing the role of pre-training data in Large Language Models (LLMs) downstream behaviour. Despite the importance, there is no public tool that supports such analysis of pre-training corpora at large scale. To help research in this space, we launch Koala, a searchable index over large pre-training corpora using compressed suffix arrays with highly efficient compression rate and search support. In its first release we index the public proportion of OPT 175B pre-training data. Koala provides a framework to do forensic analysis on the current and future benchmarks as well as to assess the degree of memorization in the output from the LLMs. Koala is available for public use at https://koala-index.erc.monash.edu/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
