arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2409.12929cs.CL
收藏

LogicPro:通过程序引导学习提升复杂逻辑推理能力

LogicPro: Improving Complex Logical Reasoning via Program-Guided Learning

Wangxuan Institute of Computer Technology, Peking University(北京大学信息科学技术研究院);College of Computer Science and Technology, Zhejiang University(浙江大学计算机科学与技术学院);Meituan Group(美团集团);Institute of Trustworthy Embodied AI, Fudan University(复旦大学可信具身人工智能研究院)

Jin Jiang, Yuchen Yan, Yang Liu, Jianing Wang, Shuai Peng, Xunliang Cai, Yixin Cao, Mengdi Zhang, Liangcai Gao

AI总结:

本研究提出LogicPro程序引导的逻辑推理数据合成方法,借助算法题与程序解生成带标准答案和推理过程的54万条数据,有效提升多个模型在多项推理基准上的表现。

摘要:

本文提出一种名为LogicPro的新型数据合成方法,该方法利用LeetCode风格的算法题目及其对应的程序解决方案,合成文本格式的复杂逻辑推理数据。首先,我们通过源算法题目和测试用例合成复杂推理问题;其次,基于标准Python解决方案与测试用例,获取每个问题的标准答案及中间变量输出;最后,在代码中间变量的引导下,为每个推理问题合成文本推理过程。通过该方法,我们能够合成难度高、可扩展、效果好,且附带黄金标准答案与高质量推理过程的数据。研究结果显示,仅用2360道算法题目构建的54万条合成数据集,我们的方法(代码与数据公开于https://github.com/jiangjin1999/LogicPro)在BBH²⁷、LogicBench、DROP、AR-LSAT、GSM8K等数据集的多个模型上均取得显著提升,性能优于大量现有推理数据集。

英文摘要:

In this paper, we propose a new data synthesis method called \textbf{LogicPro}, which leverages LeetCode-style algorithm \underline{Pro}blems and their corresponding \underline{Pro}gram solutions to synthesize Complex \underline{Logic}al Reasoning data in text format. First, we synthesize complex reasoning problems through source algorithm problems and test cases. Then, standard answers and intermediate variable outputs are obtained for each problem based on standard python solutions and test cases. Finally, with the guidance of code intermediate variables, we synthesize the text reasoning process for each reasoning problems. Through this method, we can synthesize data that is difficult, scalable, effective, and comes with golden standard answers and high-quality reasoning processes. As a result, with our 540K synthesized dataset constructed solely from 2,360 algorithm problems, our approach \footnote{Code and data are publicly available at https://github.com/jiangjin1999/LogicPro} achieves significant improvements in multiple models for the datasets \textit{BBH$^{27}$}, \textit{LogicBench}, \textit{DROP}, \textit{AR-LSAT}, and \textit{GSM8K}, etc. outperforming a wide range of existing reasoning datasets.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑