通过乔姆斯基层级评估大语言模型的正式推理能力
Evaluating the Formal Reasoning Capabilities of Large Language Models through Chomsky Hierarchy
AI总结：
本文提出ChomskyBench基准，通过乔姆斯基层级系统评估大语言模型的正式推理能力，揭示其在结构化语言处理中的性能分层及效率瓶颈。
摘要：
大语言模型的正式推理能力对推动自动化软件工程至关重要。然而，现有评估基准缺乏基于计算和复杂性的系统性评估，导致对其正式推理能力的理解存在关键缺口。为此，本文引入ChomskyBench，首次结合乔姆斯基层级全覆盖、自然语言过程跟踪评估和确定性符号可验证性，设计了全面的语言识别与生成任务，以测试各层级能力。实验表明，性能分层与层级复杂性相关，任务难度增加显著影响推理长度和性能。此外，尽管大模型和先进推理方法有相对优势，但面临严重效率瓶颈：实现实用可靠性需极高的计算成本，揭示当前限制源于效率不足而非绝对能力上限。时间复杂度分析进一步表明，LLM在这些形式任务中显著不如传统算法程序高效。这些结果界定了当前LLM的实用限制，突显了传统软件工具的不可或缺性，并为未来具备更强形式推理能力的LLM发展提供了见解。
英文摘要：
The formal reasoning capabilities of LLMs are crucial for advancing automated software engineering. However, existing benchmarks for LLMs lack systematic evaluation based on computation and complexity, leaving a critical gap in understanding their formal reasoning capabilities. Therefore, it is still unknown whether SOTA LLMs can grasp the structured, hierarchical complexity of formal languages as defined by Computation Theory. To address this, we introduce ChomskyBench, a benchmark for systematically evaluating LLMs through the lens of Chomsky Hierarchy. Unlike prior work that uses vectorized classification for neural networks, ChomskyBench is the first to combine full Chomsky Hierarchy coverage, process-trace evaluation via natural language, and deterministic symbolic verifiability. ChomskyBench is composed of a comprehensive suite of language recognition and generation tasks designed to test capabilities at each level. Extensive experiments indicate a clear performance stratification that correlates with the hierarchy's levels of complexity. Our analysis reveals a direct relationship where increasing task difficulty substantially impacts both inference length and performance. Furthermore, we find that while larger models and advanced inference methods offer notable relative gains, they face severe efficiency barriers: achieving practical reliability would require prohibitive computational costs, revealing that current limitations stem from inefficiency rather than absolute capability bounds. A time complexity analysis further indicates that LLMs are significantly less efficient than traditional algorithmic programs for these formal tasks. These results delineate the practical limits of current LLMs, highlight the indispensability of traditional software tools, and provide insights to guide the development of future LLMs with more powerful formal reasoning capabilities.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
