提示的价值：一种大语言模型（LLM）相对柯尔莫哥洛夫复杂度的方法

中文摘要 AI 辅助

在有价值的产物日益由大语言模型（LLM）创建、完成或处理的当下，核心经济问题不仅是LLM能产出什么，更是我们为其提供的输入（即提示）中保留了多少“价值”。给定一段提示、提示词、批评意见、问题陈述或部分解决方案，它们可辅助LLM生成产物z（如证明、程序、设计或科学假设），应如何衡量该输入的价值？直观来看，当输入使目标产物更易被模型生成时，它便是有价值的：要么提升其采样概率，要么减少寻找该产物所需的思考时间。我们针对该问题提出了一种基于Levin–柯尔莫哥洛夫复杂度的计算方法，将经典定义中的通用图灵机替换为LLM本身。具体而言，我们引入了一种LLM相对的“概率性Levin–柯尔莫哥洛夫复杂度”（记为pKt），将模型的思考过程视为程序的随机纸带，并以Levin的方式对其取对数计费；同时将提示价值定义为相对于pKt的算法互信息。这一方法契合上述直观认知：若某提示对产物z具有b位价值，则z会“更易获得”2^b倍，具体体现为将成功概率乘以2^b、将所需计算量除以2^b，或概率与计算量之间的任何对应权衡。与经典的算法互信息概念不同，我们提出的方法可被高效估算。我们还证明，在一项自然的复制实验中，b位的提示价值意味着，在无该提示时复制z的中位数token成本是使用该提示时的2^b倍。

英文摘要

In a world where valuable artifacts are increasingly created, completed, or processed by LLMs, the central economic question is not only what the LLM can produce, but what \emph{value} remains in the inputs (i.e., the prompts) we provide to it. Given a prompt, hint, critique, problem statement, or partial solution that helps an LLM produce an artifact $z$---a proof, program, design, or scientific hypothesis---how should we measure the value of that input? Intuitively, an input is valuable when it makes the target artifact easier for the model to generate: either by increasing its sampling probability, or by reducing the thinking time needed to find it. We propose a computational Levin--Kolmogorov complexity approach to this problem, by appropriately replacing the universal Turing machine in the classical definitions by the LLM itself. Concretely, we introduce an LLM-relative notion of \emph{probabilistic Levin--Kolmogorov complexity} $pKt$---treating the model's thinking as the random tape of the program, and charging logarithmically for it in Levin's manner---and define prompt value as algorithmic mutual information with respect to $pKt$. This captures the intuition above: a prompt having $b$ bits of value for an artifact $z$ makes $z$ $2^b$ times ``easier to obtain'', by multiplying the success probability by $2^b$, by dividing the required computation by $2^b$, or by any corresponding tradeoff between probability and computation. In contrast to the classical notion of algorithmic mutual information, ours is efficiently estimable. We additionally show that, under a natural reproduction experiment, a prompt value of \(b\) bits means that reproducing \(z\) without the prompt has median token cost \(2^b\) times that of reproducing it with the prompt.

发表机构

• Cornell Tech（康奈尔科技学院）
• Technion（以色列理工学院）
• TAU（特拉维夫大学）

机构由 AI 辅助整理，请以论文原文为准。