数据相似性不足以解释语言模型性能
Data Similarity is Not Enough to Explain Language Model Performance
AI总结：
该研究通过大规模比较Pile和C4预训练数据与下游基准，发现数据相似性度量仅与多语言任务性能相关，在其他基准中不相关，揭示预训练与下游任务关系比假设更复杂。
摘要：
大型语言模型在许多下游任务上取得了高性能，但并非所有任务都如此。通常认为，预训练数据与任务数据之间的交互决定了这种差异：一个任务的数据与模型预训练数据越相似，该任务对该模型就越容易。我们通过大规模比较Pile和C4预训练数据集与下游基准，检验了分布级和示例级相似性度量（基于嵌入、基于词元和基于模型）是否与语言模型性能相关。对于多语言数据集，相似性与性能相关，但在其他基准中，我们惊讶地发现相似性度量与准确率甚至彼此之间均不相关。这表明预训练数据与下游任务之间的关系比通常假设的更为复杂。
英文摘要：
Large language models achieve high performance on many but not all downstream tasks. The interaction between pretraining data and task data is commonly assumed to determine this variance: a task with data that is more similar to a model's pretraining data is assumed to be easier for that model. We test whether distributional and example-specific similarity measures (embedding-, token- and model-based) correlate with language model performance through a large-scale comparison of the Pile and C4 pretraining datasets with downstream benchmarks. Similarity correlates with performance for multilingual datasets, but in other benchmarks, we surprisingly find that similarity metrics are not correlated with accuracy or even each other. This suggests that the relationship between pretraining data and downstream tasks is more complex than often assumed.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
