arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2604.07569cs.LGcs.AIcs.CLcs.ITmath.IT
收藏

学习是遗忘:LLM训练作为有损压缩

Learning is Forgetting: LLM Training As Lossy Compression

Princeton University(普林斯顿大学);Cohere

Henry C. Conklin, Tom Hosking, Tan Yi-Chern, Julian Gold, Jonathan D. Cohen, Thomas L. Griffiths, Max Bartolo, Seraphina Goldfarb-Tarrant

AI总结:

本文提出LLM训练可视为有损压缩过程,通过保留训练数据中与目标相关的信息,揭示模型表示结构与下游性能的联系。

摘要:

尽管大型语言模型(LLMs)日益普及,但对其表示空间结构的理解仍有限。本文认为LLM最佳视为有损压缩实例,训练过程中仅保留与目标相关的信息。通过预训练结果展示,最优压缩模型接近信息瓶颈界限。不同模型因数据和训练方法差异而压缩方式不同,但模型的压缩最优性和信息内容可预测下游性能,从而将表示结构与模型性能联系起来。本文为大规模部署提供了统一的信息论框架。

英文摘要:

Despite the increasing prevalence of large language models (LLMs), we still have a limited understanding of how their representational spaces are structured. This limits our ability to interpret how and what they learn or relate them to learning in humans. We argue LLMs are best seen as an instance of lossy compression, where over training they learn by retaining only information in their training data relevant to their objective(s). We show pre-training results in models that are optimally compressed for next-sequence prediction, approaching the Information Bottleneck bound on compression. Across an array of open weights models, each compresses differently, likely due to differences in the data and training recipes used. However even across different families of LLMs the optimality of a model's compression, and the information present in it, can predict downstream performance on across a wide array of benchmarks, letting us directly link representational structure to actionable insights about model performance. In the general case the work presented here offers a unified Information-Theoretic framing for how these models learn that is deployable at scale.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑