arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2406.12172cs.AI
收藏

Navigating the Labyrinth: 评估 LLMs 对搜索问题的推理能力

Navigating the Labyrinth: Evaluating LLMs' Ability to Reason About Search Problems

Berkeley AI Research, UC Berkeley(加州大学伯克利分校伯克利人工智能研究所);MIT-IBM Watson AI Lab(麻省理工学院-IBM沃森人工智能实验室)

Nasim Borazjanizadeh, Roei Herzig, Trevor Darrell, Rogerio Feris, Leonid Karlinsky

AI总结:

针对 LLMs 在搜索类逻辑问题上表现不佳的挑战,研究提出 SearchBench 基准,发现提示模型生成完整 A* 算法并配合 MSMT 推理方法可显著提升其解题正确率至 57% 以上。

摘要:

大型语言模型(LLMs)近期在数学和推理基准测试中取得了令人瞩目的成绩。然而,它们在人类相对容易解决的逻辑问题和谜题上经常遇到困难。为了进一步调查这一点,我们引入了一个新基准 SearchBench,包含 11 个受直觉谜题启发的独特搜索问题。每种 SearchBench 问题类型都配备了自动化流水线,可生成任意数量的实例,并分析 LLM 生成解的可行性、正确性和最优性。我们表明,使用逐步的纯语言推理,即使是最先进的 LLMs 也无法解决 SearchBench;例如,OpenAI 的前沿模型 GPT-4 和 o1-preview 分别仅解决了 1.4% 和 18.6% 的问题。原因是 SearchBench 问题需要考虑多条路径并执行回溯,对自回归模型构成了重大挑战。有趣的是,当我们提示模型生成完整的 A* 搜索算法(一项认知难度相对更高的任务)时,性能显著提升。这种方法有效地将迭代搜索和回溯过程从模型中卸载,而这些过程在文本中是它们难以处理的。该上下文学习基线通过 Multi-Stage-Multi-Try (MSMT) 推理方法进一步增强,将 GPT-4 的正确解比例提高到 57% 以上。

英文摘要:

Large Language Models (LLMs) have recently achieved impressive performance in math and reasoning benchmarks. However, they often struggle with logic problems and puzzles that are relatively easy for humans. To further investigate this, we introduce a new benchmark, SearchBench, which contains 11 unique search problems inspired by intuitive puzzles. Each SearchBench problem type is equipped with automated pipelines to generate an arbitrary number of instances and analyze the feasibility, correctness, and optimality of LLM-generated solutions. We show that using step-by-step, language-only reasoning, even the most advanced LLMs fail to solve SearchBench; for example, OpenAI's frontier models GPT-4 and o1-preview solve only 1.4% and 18.6% of problems, respectively. The reason is that SearchBench problems require considering multiple pathways and performing backtracking, posing a significant challenge to auto-regressive models. Interestingly, performance is significantly boosted when we prompt models to generate a complete A* search algorithm - a comparatively more cognitively difficult task. This approach effectively offloads the iterative search and backtracking process from the models, which they struggle with in text. This in-context learning baseline is further enhanced via a Multi-Stage-Multi-Try (MSMT) inference method, increasing GPT-4's rate of correct solutions to over 57%.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑