Scheherazade：通过问题链评估大语言模型中的链式思维数学推理
Scheherazade: Evaluating Chain-of-Thought Math Reasoning in LLMs with Chain-of-Problems
AI总结：
提出自动化基准生成方法Scheherazade，通过前向和后向链式连接问题生成数学推理基准，评估显示o1-preview在链式推理中性能持续，优于其他前沿模型。
摘要：
基准测试对于衡量大语言模型（LLM）的推理能力至关重要。一些基准甚至已成为此类能力的实际指标。然而，随着LLM推理能力的提升，现有广泛使用的基准（如GSM8K）仅能勉强捕捉模型推理差异——例如，大多数最先进的模型在GSM8K数据集上的准确率超过94%（paperwithcode, 2024）。虽然构建更难的基准是可能的，但其创建通常是手动的、昂贵的且不可扩展的。因此，我们提出Scheherazade，一种自动化方法，通过逻辑地链式连接一小部分起始问题来生成大量具有挑战性的数学推理基准。我们提出了两种不同的链式方法：前向链式和后向链式，它们包含随机分支技术以生成复杂的推理问题。我们将Scheherazade应用于GSM8K以创建GSM8K-Scheherazade，并评估了3个前沿LLM和OpenAI的o1-preview。我们表明，虽然其他前沿模型的性能在仅链式连接几个问题时急剧下降，但我们的评估表明o1-preview的性能持续，旗舰OpenAI模型是唯一在后向推理中表现更好的模型。我们的数据和代码可在https://github.com/YoshikiTakashima/scheherazade-code-data获取。
英文摘要：
Benchmarks are critical for measuring Large Language Model (LLM) reasoning capabilities. Some benchmarks have even become the de facto indicator of such capabilities. However, as LLM reasoning capabilities improve, existing widely-used benchmarks such as GSM8K marginally encapsulate model reasoning differentials - most state-of-the-art models for example achieve over 94% accuracy on the GSM8K dataset (paperwithcode, 2024). While constructing harder benchmarks is possible, their creation is often manual, expensive, and unscalable. As such, we present Scheherazade, an automated approach to produce large quantities of challenging mathematical reasoning benchmarks by logically chaining a small starting set of problems. We propose two different chaining methods, forward chaining and backward chaining, which include randomized branching techniques to generate complex reasoning problems. We apply Scheherazade on GSM8K to create GSM8K-Scheherazade and evaluate 3 frontier LLMs and OpenAI's o1-preview on it. We show that while other frontier models' performance declines precipitously at only a few questions chained, our evaluation suggests o1-preview's performance persists, with the flagship OpenAI model the only one to perform better at backward reasoning. Our data and code are available at https://github.com/YoshikiTakashima/scheherazade-code-data.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
