程序推理的错觉：衡量大语言模型中的长时序有限状态机执行能力
The Illusion of Procedural Reasoning: Measuring Long-Horizon FSM Execution in LLMs
AI总结：
本研究提出以有限状态机（FSM）执行作为可控基准，探测大语言模型的长时序程序推理能力，发现模型性能随时序长度、分支复杂度增加系统性下降，大模型仅局部准确率提升，需外化中间步骤才能改善多步推理脆性。
摘要：
大语言模型（LLM）在被归类为推理问题的任务上已取得显著成果，但其执行程序推理——即开展多步骤、基于规则的计算——的真实能力仍不明确。与能够确定性执行长时序符号程序的算法系统不同，LLM在推理链延长时性能往往会下降，但目前尚无可控、可解释的基准来分离并衡量这种性能崩塌。\n我们提出将有限状态机（Finite-State Machine, FSM）执行作为评估LLM程序推理能力的最小化、完全可解释的框架。在我们的实验设置中，模型会获得明确的FSM定义，必须根据输入动作逐步执行该状态机，并在多轮交互中保持状态一致性。该任务无需世界知识，仅要求忠实地应用确定性转移规则，因此可直接探测模型内部的程序执行保真度。我们同时测量轮次准确率（Turn Accuracy）和任务准确率（Task Accuracy），以区分即时计算能力与累积状态维持能力。\n实验结果显示，随着任务时序长度或分支复杂度增加，模型性能会出现系统性下降。与记忆跨度较长的情况相比，当规则检索涉及高分支因子时，模型的表现要差得多。更大的模型局部准确率有所提升，但在多步推理下仍然脆弱，除非通过显式提示让其外化中间步骤。基于FSM的评估提供了一种透明、复杂度可控的探测方式，可用于诊断这种失效模式，并指导归纳偏置的设计，以实现真正的长时序程序执行能力。通过将推理建立在可测量的执行保真度而非表面正确性之上，本研究为理解和提升LLM的算法可靠性奠定了严谨的实验基础。
英文摘要：
Large language models (LLMs) have achieved remarkable results on tasks framed as reasoning problems, yet their true ability to perform procedural reasoning, executing multi-step, rule-based computations remains unclear. Unlike algorithmic systems, which can deterministically execute long-horizon symbolic procedures, LLMs often degrade under extended reasoning chains, but there is no controlled, interpretable benchmark to isolate and measure this collapse. We introduce Finite-State Machine (FSM) Execution as a minimal, fully interpretable framework for evaluating the procedural reasoning capacity of LLMs. In our setup, the model is given an explicit FSM definition and must execute it step-by-step given input actions, maintaining state consistency over multiple turns. This task requires no world knowledge, only faithful application of deterministic transition rules, making it a direct probe of the model's internal procedural fidelity. We measure both Turn Accuracy and Task Accuracy to disentangle immediate computation from cumulative state maintenance. Empirical results reveal systematic degradation as task horizon or branching complexity increases. Models perform significantly worse when rule retrieval involves high branching factors than when memory span is long. Larger models show improved local accuracy but remain brittle under multi-step reasoning unless explicitly prompted to externalize intermediate steps. FSM-based evaluation offers a transparent, complexity-controlled probe for diagnosing this failure mode and guiding the design of inductive biases that enable genuine long-horizon procedural competence. By grounding reasoning in measurable execution fidelity rather than surface correctness, this work helps establish a rigorous experimental foundation for understanding and improving the algorithmic reliability of LLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
