arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.21061cs.AI
收藏

工具构建作为通往‘超级智能’的路径

Tool Use Reduces Depth-Induced Collapse in OOD Reasoning

David Koplow(独立研究者);Tomer Galanti(独立研究者);Tomaso Poggio(独立研究者)

David Koplow, Tomer Galanti, Tomaso Poggio

AI总结:

本文通过设计基准测试,探讨LLMs在逻辑推理任务中通过工具调用实现超级智能的潜力。

摘要:

Diligent Learner框架建议,只要具备足够的步骤成功概率γ,LLMs可以通过测试时搜索实现超级智能。本文设计了一个基准测试来衡量LLMs在逻辑离分布推断中的γ值。我们构建了一类涉及GF(2)电路重建的任务,这些任务随着推理步骤的增加而变得更加困难,并且从信息论的角度来看,除非LLM仔细整合所有提供的信息,否则无法可靠地解决。我们的分析表明,尽管小型LLMs的γ值随着深度增加而超线性下降,但前沿模型在此任务上表现出部分鲁棒性。此外,我们发现大规模成功推理依赖于精确的工具调用,识别工具设计是LLMs通过Diligent Learner框架实现一般超级智能的关键能力。

英文摘要:

Humans can apply ideas learned in one context to substantially different situations. We call this process of searching for and constructing novel recombinations of learned relationships to solve new problems \textit{out-of-distribution (OOD) reasoning}. The capacity for large language models (LLMs) to support OOD reasoning underpins proposals for generally intelligent systems. However, this property is challenging to measure because most problems admit many decompositions, some involving shallow subproblems and others involving subproblems that may have been memorized from the training data. This makes it difficult to determine how much compositional reasoning a model must actually perform. Uncertainty about training distributions, how to measure a datapoint's distance from a training distribution, and the exponential number of ways to decompose most tasks make it intractable to robustly measure any modern LLM's OOD-reasoning capacity on standard natural-language tasks. In this work, we introduce a benchmark in which a model progressively solves a Boolean circuit over $GF(2)$ from data. This benchmark is both minimal, isolating OOD reasoning from these confounding factors, and general, as any computable function can be represented as a sufficiently large $GF(2)$ polynomial. We find that standalone models' next-step accuracy collapses as depth grows. In contrast, tool use through the synthesis and execution of code prevents this collapse in both small and frontier LLMs. These results indicate that synthesizing tools play a crucial role in supporting OOD reasoning.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑