当工具和规划如何帮助大语言模型思考？一个成本和延迟感知的基准测试
When Do Tools and Planning Help Large Language Models Think? A Cost- and Latency-Aware Benchmark
AI总结：
研究通过对比不同配置评估工具和规划对大语言模型性能的影响，发现工具增强能显著提升准确性但增加延迟，而复杂工具协调易导致模型退化。
摘要：
现代大语言模型（LLMs）越来越多地依赖推理时间的规划和外部工具来提高推理能力。我们在此两个现实场景上进行了基准测试：基于图结构知识的事件中心问答（Event-QA）和Reddit ChangeMyView（CMV）中的说服性回应生成。使用LangChain和LangGraph，我们比较了一个单次基线与配备任务特定工具（DBpedia SPARQL/查找/模式探索、维基百科聚焦检索和主题网络搜索）的计划-执行-重新计划代理。我们在Event-QA和CMV中各评估60个示例（3个20个的分割），并报告了平均端到端延迟和每例标记成本估计。我们评估了GPT-4o和GPT-4o-mini在相同的工作流下，报告了准确率和端到端延迟。在Event-QA中，最佳工具增强配置提高了准确性（例如，GPT-4o的47.5%→67.5%），但延迟增加了多个数量级（≈8秒→≈317秒每例）。在CMV中，单次提示最强（例如，GPT-4o-mini在≈6秒时达到75%），而规划+搜索显著增加延迟但没有一致的收益。然而，复杂的多工具协调暴露了失败模式，其中较小的模型会退化。总体而言，发现突显了在模型大小和代理/工具复杂性方面需要任务特定的、成本感知的选择。
英文摘要：
Modern large language models (LLMs) increasingly rely on inference-time planning and external tools to improve reasoning. We benchmark this behavior on two real-world settings: event-centric question answering over graph-structured knowledge (Event-QA) and persuasive response generation in Reddit ChangeMyView (CMV). Using LangChain and LangGraph, we compare a one-shot baseline against a plan-execute-replan agent equipped with task-specific tools (DBpedia SPARQL/lookup/schema exploration, Wikipedia-focused retrieval, and topical web search). We evaluate on 60 examples each from Event-QA and CMV (3 splits of 20), and report both mean end-to-end latency and per-example token cost estimates. We evaluate GPT-4o and GPT-4o-mini under identical workflows and report accuracy and end-to-end latency. On Event-QA, the best tool-augmented configuration improves accuracy (e.g., 47.5\% $\rightarrow$ 67.5\% for GPT-4o) while increasing latency by orders of magnitude ($\sim$8s $\rightarrow$ $\sim$317s per example). On CMV, one-shot prompting is strongest (e.g., GPT-4o-mini achieves 75\% at $\sim$6s), and planning+search increases latency substantially without consistent gains. However, complex multi-tool orchestration exposes failure modes where the smaller model degrades. Overall, the findings highlight the need for task-specific, cost-aware choices of both model size and agent/tooling complexity.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
