ItinBench：利用大语言模型在多个认知维度上进行规划的基准测试
ItinBench: Benchmarking Planning Across Multiple Cognitive Dimensions with Large Language Models
AI总结：
本文提出ItinBench，通过整合空间推理和传统语言推理任务，评估大语言模型在多认知维度上的规划能力，发现模型在同时处理多个维度时性能不稳定。
摘要：
大型语言模型（LLMs）凭借先进的认知能力正逐渐成为各类推理和规划任务的代理。传统评估往往聚焦于特定推理或规划问题在内的受控环境。近期研究探索了旅行规划作为整合各种语言推理任务到现实世界情境的媒介。然而，推理任务超越了单纯的语言推理，全面评估LLMs需要一个整合多个认知领域的测试平台。为解决这一缺口，我们引入ItinBench，一个包含空间推理任务（即路线优化）的旅行行程规划任务，同时保留传统语言推理任务。ItinBench在多样化任务上同时评估各种LLMs，包括Llama 3.1 8B、Mistral Large、Gemini 1.5 Pro和GPT家族。我们的发现揭示了LLMs在同时处理多个认知维度时难以保持高性能和一致性。通过整合不同人类水平认知领域的任务，ItinBench为构建更全面的推理测试平台提供了新见解，这些平台更能反映现实世界挑战。代码和数据集：https://ethanwtl.github.io/IBweb/
英文摘要：
Large language models (LLMs) with advanced cognitive capabilities are emerging as agents for various reasoning and planning tasks. Traditional evaluations often focus on specific reasoning or planning questions within controlled environments. Recent studies have explored travel planning as a medium to integrate various verbal reasoning tasks into real-world contexts. However, reasoning tasks extend beyond verbal reasoning alone, and a comprehensive evaluation of LLMs requires a testbed that incorporates tasks from multiple cognitive domains. To address this gap, we introduce ItinBench, a benchmark that features one task of spatial reasoning, i.e., route optimization, into trip itinerary planning while keeping the traditional verbal reasoning tasks. ItinBench evaluates various LLMs across diverse tasks simultaneously, including Llama 3.1 8B, Mistral Large, Gemini 1.5 Pro, and GPT family. Our findings reveal that LLMs struggle to maintain high and consistent performance when concurrently handling multiple cognitive dimensions. By incorporating tasks from distinct human-level cognitive domains, ItinBench provides new insights into building more comprehensive reasoning testbeds that better reflect real-world challenges. The code and dataset: https://ethanwtl.github.io/IBweb/
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
