从条件柯尔莫哥洛夫复杂度视角统一两类缩放定律
Unifying Two Types of Scaling Laws from the Perspective of Conditional Kolmogorov Complexity
AI总结：
本文从条件柯尔莫哥洛夫复杂度视角分析LLMs训练与推理过程，统一了OpenAI 2020年提出的训练相关缩放定律和2024年提出的推理相关缩放定律，揭示两类定律均通过增加图灵机执行步骤提升复杂度近似程度。
摘要：
2020年，OpenAI提出第一类缩放定律，描述模型损失与参数、数据及训练计算规模之间的关系；2024年，该机构提出第二类缩放定律，描述模型推理性能与推理计算之间的关系。本文从无损压缩视角出发，利用条件柯尔莫哥洛夫复杂度分析大语言模型（LLMs）的训练和推理过程，统一了这两类缩放定律。研究发现，两类缩放定律均通过增加图灵机的执行步骤来提升对条件柯尔莫哥洛夫复杂度的近似程度：第一类通过增加模型参数数量实现，第二类则通过增加中间标记数量实现。
英文摘要：
In 2020, OpenAI proposed the first type of Scaling Laws, describing the relationships between model loss and the scale of parameters, data, and training computation. In 2024, OpenAI proposed the second type of Scaling Laws, describing the relationship between model inference performance and inference computation. In this paper, we analyze LLMs training and inference processes from the perspective of lossless compression using conditional Kolmogorov complexity, and unify these two types of Scaling Laws. We find that both types of Scaling Laws improve approximation of conditional Kolmogorov complexity by increasing execution steps of Turing machine. The first type of Scaling Laws increases execution steps by increasing number of model parameters. The second type of Scaling Laws increases execution steps by increasing the number of intermediate tokens.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
