思维程序何时对推理有效？
When Do Program-of-Thoughts Work for Reasoning?
AI总结：
本文提出 CIRS 衡量代码结构与逻辑复杂度对 LLM 推理能力的影响，并设计自动合成分层算法用于数学指令生成和代码数据过滤。
摘要：
在具身人工智能领域，大型语言模型（LLM）的推理能力发挥着关键作用。尽管已有 program-of-thought prompting（思维程序提示）等有效方法，利用编程语言帮助 LLM 处理复杂推理任务，但代码数据对推理能力提升的具体影响仍未得到充分研究。为弥补这一空白，我们提出 complexity-impacted reasoning score（CIRS，复杂度影响推理分数），该分数结合结构属性与逻辑属性，用于衡量代码与推理能力之间的相关性。具体而言，我们使用抽象语法树编码结构信息，并通过综合考虑题目难度和圈复杂度来计算逻辑复杂度。通过实证分析，我们发现并非所有复杂度的代码数据都能被 LLM 学习或理解；最佳复杂度水平对于借助程序辅助提示提升推理能力至关重要。随后，我们设计了一种自动合成与分层算法，并将其应用于数学推理的指令生成以及代码生成任务的代码数据过滤。大量结果证明了所提方法的有效性。代码将集成到 EasyInstruct 框架中，地址为 https://github.com/zjunlp/EasyInstruct。
英文摘要：
In the realm of embodied artificial intelligence, the reasoning capabilities of Large Language Models (LLMs) play a pivotal role. Although there are effective methods like program-of-thought prompting for LLMs which uses programming language to tackle complex reasoning tasks, the specific impact of code data on the improvement of reasoning capabilities remains under-explored. To address this gap, we propose complexity-impacted reasoning score (CIRS), which combines structural and logical attributes, to measure the correlation between code and reasoning abilities. Specifically, we use the abstract syntax tree to encode the structural information and calculate logical complexity by considering the difficulty and the cyclomatic complexity. Through an empirical analysis, we find not all code data of complexity can be learned or understood by LLMs. Optimal level of complexity is critical to the improvement of reasoning abilities by program-aided prompting. Then we design an auto-synthesizing and stratifying algorithm, and apply it to instruction generation for mathematical reasoning and code data filtering for code generation tasks. Extensive results demonstrates the effectiveness of our proposed approach. Code will be integrated into the EasyInstruct framework at https://github.com/zjunlp/EasyInstruct.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
