人工智能的首次检验：一种基于大语言模型（LLM）参考的超智能C测试
Artificiality's First Exam: An LLM-Referenced C-test for Supraintelligence
AI总结：
本文提出首个基于Levin的Kt、以LLM为参考机器的超智能C测试，设计了生成验证流程并在12个LLM上验证，可生成超出前沿智能体能力的高难度测试项。
摘要：
在AI发展加速的背景下，任何以人类或当前AI系统难度校准的基准都会陷入循环，或很快达到饱和。那么，我们该如何评估可能比当前机器和编写测试的人更强大的机器？该问题的一个答案依赖于逆问题，即给定观测结果和先验知识，难度取决于底层机制复杂性的归纳推理问题。Levin的Kt（柯尔莫哥洛夫复杂度的可计算变体）为该难度设定了理论上界，平衡了对机制的最简表示编码与运行该机制的时间步长的对数。此前基于Levin的Kt的测试（如原始C测试）基于极简通用图灵机（UTM），其表示能力极为有限且无先验知识。推理型大语言模型（LLM）是捕获人类文化知识纲要的通用图灵机，将提示词作为自然语言程序使用。本文提出首个以LLM作为表示机器、基于Levin的Kt的测试。在任何人类或前沿模型解决这些测试项之前，我们可以生成高难度的全新测试项，其难度以比特为单位的比率尺度衡量。我们对该测度进行了形式化，设计了轻量级的生成与验证流程，探究了参考机器的能力以及测试者与被测试者之间的差距的影响。我们生成了首批经验证的测试项，并将其应用于12个LLM。所得智能体特征曲线随难度降低，延伸至未知的高难度区域——即我们可验证的测试项的超智能窗口，该窗口超出了前沿智能体的能力范围。
英文摘要：
Under the accelerated pace of AI development, any benchmark whose difficulty is calibrated with humans or current AI systems becomes circular, or saturates soon. How can we then evaluate machines that may become more capable than the current machines and the people who write the tests? One answer to the question hinges on inverse problems, inductive inference questions for which difficulty depends on the complexity of the underlying mechanism, given the observations and the previous knowledge. Levin's Kt, a computable variant of Kolmogorov complexity, puts a theoretical upper bound to that difficulty, balancing the simplest representation encoding for the mechanism and the logarithm of the time steps to run that mechanism. Previous tests derived from Levin's Kt, such as the original C-test, are based on minimalistic Universal Turing Machines (UTM), capable of very limited representations and no previous knowledge. Reasoning large language models (LLM) are UTMs that capture a compendium of human culture as knowledge, using prompts as natural language programs. This paper presents the first test based on Levin's Kt with LLMs as representational machines. Fresh test items can be produced at high difficulties, estimated in a ratio scale measured in bits, before any human or frontier model can solve them. We formalise the measure, devise a lightweight generation and certification pipeline, explore the effect of the power of the reference machine and the gap between examiner and examinee. We produce a first battery of certified items administered to twelve LLMs. The resulting agent characteristic curves decrease with difficulty, down to the uncharted high-difficulty region, the supra-intelligence window of items we can certify, yet beyond generations of frontier intelligence.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
