预训练暴露解释了大型语言模型中的流行度判断
Pretraining Exposure Explains Popularity Judgments in Large Language Models
AI总结：
研究通过分析预训练数据中的暴露统计，发现大型语言模型对流行实体的偏好主要受预训练暴露影响，而非外部流行度信号，揭示了数据暴露在驱动流行度偏差中的核心作用。
摘要：
大型语言模型（LLMs）表现出对知名实体的系统性偏好，这种现象通常归因于流行度偏差。然而，这些偏好是反映真实世界流行度还是预训练期间的统计暴露仍不明确，主要由于训练语料的不可访问性。本文提供了首个基于完全可观察预训练数据的直接大规模流行度偏差分析。利用开放的OLMo模型及其完整预训练语料Dolma，我们计算了7.4万亿token级别的精确实体暴露统计。我们分析了涵盖五类（人物、地点、组织、艺术、产品）的2000个实体，并将预训练暴露与维基百科页面浏览量以及两种 elicited LLM 流行度信号（直接标量估计和配对比较）进行比较。我们的结果表明，预训练暴露与维基百科流行度高度相关，验证了暴露作为训练期间真实世界显著性代理的有效性。更重要的是，我们发现LLM流行度判断更接近暴露而非维基百科，尤其是在通过配对比较 elicited 时。这种一致性在较大模型中最强，并在长尾部分持续存在，其中维基百科流行度变得不可靠。总体而言，我们的发现表明，LLM中的流行度先验主要由预训练统计塑造，而非外部流行度信号，提供了实证证据表明数据暴露在驱动流行度偏差中起核心作用。
英文摘要：
Large language models (LLMs) exhibit systematic preferences for well-known entities, a phenomenon often attributed to popularity bias. However, the extent to which these preferences reflect real-world popularity versus statistical exposure during pretraining remains unclear, largely due to the inaccessibility of most training corpora. We provide the first direct, large-scale analysis of popularity bias grounded in fully observable pretraining data. Leveraging the open OLMo models and their complete pretraining corpus, Dolma, we compute precise entity-level exposure statistics across 7.4 trillion tokens. We analyze 2,000 entities spanning five types (Person, Location, Organization, Art, Product) and compare pretraining exposure against Wikipedia pageviews and two elicited LLM popularity signals: direct scalar estimation and pairwise comparison. Our results show that pretraining exposure strongly correlates with Wikipedia popularity, validating exposure as a meaningful proxy for real-world salience during the training period. More importantly, we find that LLM popularity judgments align more closely with exposure than with Wikipedia, especially when elicited via pairwise comparisons. This alignment is strongest for larger models and persists in the long tail, where Wikipedia popularity becomes unreliable. Overall, our findings demonstrate that popularity priors in LLMs are primarily shaped by pretraining statistics rather than external popularity signals, offering concrete evidence that data exposure plays a central role in driving popularity bias.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
