代码提示激发文本+代码大语言模型的条件推理能力
Code Prompting Elicits Conditional Reasoning Abilities in Text+Code LLMs
AI总结：
本文提出代码提示方法，将自然语言问题转化为代码以激发文本+代码LLMs的条件推理能力，并在多个模型和数据集上验证了显著性能提升及其作用机制。
摘要：
推理是语言理解的基本组成部分。近期的提示技术（如思维链）在各种推理任务上持续提升了大语言模型（LLMs）的表现。然而，在推理阶段究竟是什么触发LLMs的推理能力，目前仍知之甚少。本文提出了代码提示（code prompting），这是一种提示链方法，将自然语言问题转化为代码，并直接使用生成的代码提示LLM，而不借助外部代码执行。我们假设代码提示能够激发在文本和代码上训练的LLMs的某些推理能力，并利用所提出的方法来提升条件推理能力，即根据不同条件的满足情况推断出不同结论的能力。我们发现，代码提示在多个条件推理数据集上为多个LLMs带来了显著的性能提升（GPT 3.5上最高提升22.52个百分点，Mixtral上提升7.75个百分点，Mistral上提升16.78个百分点）。随后，我们进行了全面的实验，以理解代码提示如何触发推理能力，以及底层模型中被激发了哪些能力。我们对GPT 3.5的分析表明，输入问题的代码格式对于性能提升至关重要。此外，代码提示提高了上下文学习的样本效率，并促进了对变量或实体的状态跟踪。
英文摘要：
Reasoning is a fundamental component of language understanding. Recent prompting techniques, such as chain of thought, have consistently improved LLMs' performance on various reasoning tasks. Nevertheless, there is still little understanding of what triggers reasoning abilities in LLMs in the inference stage. In this paper, we introduce code prompting, a chain of prompts that transforms a natural language problem into code and directly prompts the LLM using the generated code without resorting to external code execution. We hypothesize that code prompts can elicit certain reasoning capabilities of LLMs trained on text and code and utilize the proposed method to improve conditional reasoning, the ability to infer different conclusions depending on the fulfillment of certain conditions. We find that code prompting exhibits a high-performance boost for multiple LLMs (up to 22.52 percentage points on GPT 3.5, 7.75 on Mixtral, and 16.78 on Mistral) across multiple conditional reasoning datasets. We then conduct comprehensive experiments to understand how code prompts trigger reasoning abilities and which capabilities are elicited in the underlying models. Our analysis of GPT 3.5 reveals that the code formatting of the input problem is essential for performance improvement. Furthermore, code prompts improve sample efficiency of in-context learning and facilitate state tracking of variables or entities.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
