幻觉是空间最优性的结果：成员测试的率失真定理
Hallucination is a Consequence of Space-Optimality: A Rate-Distortion Theorem for Membership Testing
AI总结：
通过将幻觉形式化为成员测试问题，建立率失真定理，证明在有限容量下信息论最优策略必然导致对某些非事实的高置信度，从而产生幻觉。
摘要：
大型语言模型通常对缺乏可推断模式的“随机事实”以高置信度产生幻觉。我们将此类事实的记忆形式化为一个成员测试问题，统一了布隆过滤器的离散误差指标与LLM的连续对数损失。通过分析在事实在可能主张的宇宙中稀疏的情况下，我们建立了一个率失真定理：最优记忆效率由事实与非事实得分分布之间的最小KL散度刻画。这一理论框架在理想化设置下为幻觉提供了独特的解释：即使有最优训练、完美数据和简化的“封闭世界”设置，有限容量下信息论最优策略不是放弃或遗忘，而是对某些非事实赋予高置信度，从而导致幻觉。我们在合成数据和真实数据上实证验证了这一理论，表明幻觉作为有损压缩的自然结果持续存在。同一定理恢复并锐化了布隆型滤波器的经典空间下界，确定了两侧滤波器遗留的加性常数。
英文摘要：
Large language models often hallucinate with high confidence on "random facts" that lack inferable patterns. We formalize the memorization of such facts as a membership testing problem, unifying the discrete error metrics of Bloom filters with the continuous log-loss of LLMs. By analyzing this problem in the regime where facts are sparse in the universe of plausible claims, we establish a rate-distortion theorem: the optimal memory efficiency is characterized by the minimum KL divergence between score distributions on facts and non-facts. This theoretical framework provides a distinctive explanation for hallucination under an idealized setting: even with optimal training, perfect data, and a simplified ``closed world'' setting, the information-theoretically optimal strategy under limited capacity is not to abstain or forget, but to assign high confidence to some non-facts, resulting in hallucination. We validate this theory empirically on both synthetic and real-world data, showing that hallucinations persist as a natural consequence of lossy compression. The same theorem recovers and sharpens classical space lower bounds for Bloom-type filters, pinning down an additive constant left open for two-sided filters.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
