揭示通过假设分解与修正实现代码推理的奥秘
Unveiling the Magic of Code Reasoning through Hypothesis Decomposition and Amendment
AI总结：
本文提出代码推理任务及RHDA流程，通过假设分解与修正缓解多步推理中的逻辑链崩溃，性能提升达3倍，并应用于VirtualHome模拟。
摘要：
推理能力是大语言模型（LLMs）中最神秘且引人入胜的方面之一。众多研究致力于探索和拓展这种推理能力的边界。然而，同时体现推理和回忆特征的任务往往被忽视。在本文中，我们引入了这样一项新颖任务——代码推理，为LLMs的推理能力提供了新的视角。我们基于既定的逻辑推理形式总结出三个元基准，并将其实例化为八个具体的基准任务。我们在这些基准上的测试表明，LLMs在识别满意的推理路径方面仍然存在困难。此外，我们提出了一种受人类复杂问题解决方法启发的新路径探索流程。这一反思性假设分解与修正（RHDA）流程包括以下迭代步骤：（1）基于观察提出潜在假设并进行分解；（2）利用工具验证假设和反思结果；（3）根据观察修正假设。我们的方法有效缓解了多步推理中因遗忘或幻觉问题导致的逻辑链崩溃，性能提升高达3倍。最后，我们将此流程扩展应用于模拟真实场景中的复杂家务任务，特别是在VirtualHome中，增强了对失败案例的处理。我们在https://github.com/TnTWoW/code_reasoning上发布了我们的代码和所有结果。
英文摘要：
The reasoning abilities are one of the most enigmatic and captivating aspects of large language models (LLMs). Numerous studies are dedicated to exploring and expanding the boundaries of this reasoning capability. However, tasks that embody both reasoning and recall characteristics are often overlooked. In this paper, we introduce such a novel task, code reasoning, to provide a new perspective for the reasoning abilities of LLMs. We summarize three meta-benchmarks based on established forms of logical reasoning, and instantiate these into eight specific benchmark tasks. Our testing on these benchmarks reveals that LLMs continue to struggle with identifying satisfactory reasoning pathways. Additionally, we present a new pathway exploration pipeline inspired by human intricate problem-solving methods. This Reflective Hypothesis Decomposition and Amendment (RHDA) pipeline consists of the following iterative steps: (1) Proposing potential hypotheses based on observations and decomposing them; (2) Utilizing tools to validate hypotheses and reflection outcomes; (3) Revising hypothesis in light of observations. Our approach effectively mitigates logical chain collapses arising from forgetting or hallucination issues in multi-step reasoning, resulting in performance gains of up to $3\times$. Finally, we expanded this pipeline by applying it to simulate complex household tasks in real-world scenarios, specifically in VirtualHome, enhancing the handling of failure cases. We release our code and all of results at https://github.com/TnTWoW/code_reasoning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
