GroundCocoa：评估语言模型中组合式与条件式推理能力的基准
GroundCocoa: A Benchmark for Evaluating Compositional & Conditional Reasoning in Language Models
AI总结：
针对LLMs在简单任务中意外出错的问题，该研究聚焦组合式与条件式推理，提出词汇多样的基准GroundCocoa（关联机票预订任务），发现最先进LLMs性能差异显著，GPT-4 Turbo准确率未超67%。
摘要：
大型语言模型（LLMs）的快速发展使其在标准基准测试中表现出色，经常超越人类表现。这推动了许多下游应用（如LLM智能体）依赖其推理能力解决复杂任务需求。然而，LLMs在简单任务和看似直接的情况下会意外出错，凸显了需要更好、更多样的评估设置来衡量其真实能力。为此，我们选择研究人类认知核心的组合式与条件式推理，引入GroundCocoa——一个词汇多样的基准，将这些推理技能与机票预订的现实问题相连。我们的任务是将详细的用户偏好与选择题形式呈现的可用航班选项对齐。结果显示，当前最先进的LLMs之间存在显著性能差异，即使是表现最佳的GPT-4 Turbo，在使用高级提示技术的情况下准确率也未超过67%。
英文摘要：
The rapid progress of large language models (LLMs) has seen them excel and frequently surpass human performance on standard benchmarks. This has enabled many downstream applications, such as LLM agents, to rely on their reasoning to address complex task requirements. However, LLMs are known to unexpectedly falter in simple tasks and under seemingly straightforward circumstances - underscoring the need for better and more diverse evaluation setups to measure their true capabilities. To this end, we choose to study compositional and conditional reasoning, two aspects that are central to human cognition, and introduce GroundCocoa - a lexically diverse benchmark connecting these reasoning skills to the real-world problem of flight booking. Our task involves aligning detailed user preferences with available flight options presented in a multiple-choice format. Results indicate a significant disparity in performance among current state-of-the-art LLMs with even the best performing model, GPT-4 Turbo, not exceeding 67% accuracy despite advanced prompting techniques.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
