arXiv 2211.125882023-10-23cs.CLcs.AI

Program of Thoughts Prompting：在数值推理任务中将计算与推理解耦

Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks

University of Waterloo（滑铁卢大学）；Vector Institute（向量研究所）；Google Research（谷歌研究院）

Wenhu Chen, Xueguang Ma, Xinyi Wang, William W. Cohen

备注：: Published at TMLR 2023

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对数值推理任务中计算与推理耦合的问题，提出Program of Thoughts (PoT)方法，利用语言模型生成程序并由外部计算机执行，在多数据集上比CoT平均提升约12%性能，结合自洽解码达到或接近SoTA。

摘要：

最近，在教导语言模型执行逐步推理以解决复杂数值推理任务方面取得了显著进展。Chain-of-thoughts prompting (CoT) 是目前这些任务中最先进的方法。CoT 使用语言模型在多步“思考”过程中同时执行推理和计算。为了将计算与推理解耦，我们提出了 `Program of Thoughts' (PoT)，它使用语言模型（主要是 Codex）将推理过程表达为程序。计算被交由外部计算机执行，该计算机运行生成的程序以得出答案。我们在五个数学应用题数据集（GSM、AQuA、SVAMP、TabMWP、MultiArith）和三个金融问答数据集（FinQA、ConvFinQA、TATQA）上，针对 few-shot 和 zero-shot 设置评估了 PoT。在 few-shot 和 zero-shot 设置下，PoT 在所有评估数据集上比 CoT 平均提升了约 12% 的性能。通过将 PoT 与 self-consistency decoding 相结合，我们能够在所有数学问题数据集上达到 SoTA 性能，并在金融数据集上接近 SoTA 性能。我们的所有数据和代码已在 Github 发布：https://github.com/wenhuchen/Program-of-Thoughts

英文摘要：

Recently, there has been significant progress in teaching language models to perform step-by-step reasoning to solve complex numerical reasoning tasks. Chain-of-thoughts prompting (CoT) is by far the state-of-art method for these tasks. CoT uses language models to perform both reasoning and computation in the multi-step `thought' process. To disentangle computation from reasoning, we propose `Program of Thoughts' (PoT), which uses language models (mainly Codex) to express the reasoning process as a program. The computation is relegated to an external computer, which executes the generated programs to derive the answer. We evaluate PoT on five math word problem datasets (GSM, AQuA, SVAMP, TabMWP, MultiArith) and three financial-QA datasets (FinQA, ConvFinQA, TATQA) for both few-shot and zero-shot setups. Under both few-shot and zero-shot settings, PoT can show an average performance gain over CoT by around 12\% across all the evaluated datasets. By combining PoT with self-consistency decoding, we can achieve SoTA performance on all math problem datasets and near-SoTA performance on financial datasets. All of our data and code are released in Github https://github.com/wenhuchen/Program-of-Thoughts

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。