多轮谜题：评估大型语言模型（LLMs）中的交互式推理与策略对话能力
Multi-Turn Puzzles: Evaluating Interactive Reasoning and Strategic Dialogue in LLMs
AI总结：
针对LLMs在现实交互任务中表现不佳的问题，本文提出含确定性评分机制的多轮谜题基准，评估模型的交互式推理等能力，揭示其不足并为相关研究提供平台。
摘要：
大型语言模型（LLMs）擅长解决陈述清晰完整的问题，但在大多数现实场景中常见的微妙环境或交互任务中往往表现不佳。这凸显了开发能有效进行逻辑一致的多轮对话、获取信息并基于不完整数据推理的LLMs的迫切需求。为此，我们引入一个新型基准，包含一系列多轮任务，每个任务旨在测试特定的推理、交互对话和信息获取能力。这些任务具有确定性评分机制，无需人工干预。在我们的基准上评估前沿模型显示出显著的提升空间。分析表明，大多数错误源于指令遵循不佳、推理失败和规划不足。该基准为了解当前LLMs处理复杂交互场景的优缺点提供了宝贵见解，并为未来旨在提升这些关键能力的研究提供了稳健平台。
英文摘要：
Large language models (LLMs) excel at solving problems with clear and complete statements, but often struggle with nuanced environments or interactive tasks which are common in most real-world scenarios. This highlights the critical need for developing LLMs that can effectively engage in logically consistent multi-turn dialogue, seek information and reason with incomplete data. To this end, we introduce a novel benchmark comprising a suite of multi-turn tasks each designed to test specific reasoning, interactive dialogue, and information-seeking abilities. These tasks have deterministic scoring mechanisms, thus eliminating the need for human intervention. Evaluating frontier models on our benchmark reveals significant headroom. Our analysis shows that most errors emerge from poor instruction following, reasoning failures, and poor planning. This benchmark provides valuable insights into the strengths and weaknesses of current LLMs in handling complex, interactive scenarios and offers a robust platform for future research aimed at improving these critical capabilities.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
