LatEval：基于横向思维谜题的不完整信息交互式LLMs评估基准
LatEval: An Interactive LLMs Evaluation Benchmark with Incomplete Information from Lateral Thinking Puzzles
AI总结：
针对LLMs横向思维能力评估的空白，提出基于横向思维谜题的交互式基准LatEval，从提问质量和信息整合两方面测试，发现包括GPT-4在内的模型均与人类存在差距。
摘要：
随着LLMs的不断演进和完善，它们被赋予了令人印象深刻的逻辑推理或垂直思维能力。但它们能否跳出思维定势？它们是否具备熟练的横向思维能力？遵循横向思维谜题的设置，我们提出了一个新的评估基准LatEval，该基准在交互式框架内评估模型的横向思维能力。在我们的基准中，我们从两个方面挑战LLMs：模型提出的问题的质量，以及模型整合信息以解决问题的能力。我们发现，几乎所有LLMs在交互过程中都难以运用横向思维。例如，即使是最先进的模型GPT-4，虽然在一定程度上展现出优势，但与人类相比仍有明显差距。该评估基准为LLMs提供了一个极具挑战性和独特性的任务，这对有效的AI助手至关重要。
英文摘要：
With the continuous evolution and refinement of LLMs, they are endowed with impressive logical reasoning or vertical thinking capabilities. But can they think out of the box? Do they possess proficient lateral thinking abilities? Following the setup of Lateral Thinking Puzzles, we propose a novel evaluation benchmark, LatEval, which assesses the model's lateral thinking within an interactive framework. In our benchmark, we challenge LLMs with 2 aspects: the quality of questions posed by the model and the model's capability to integrate information for problem-solving. We find that nearly all LLMs struggle with employing lateral thinking during interactions. For example, even the most advanced model, GPT-4, exhibits the advantage to some extent, yet still maintain a noticeable gap when compared to human. This evaluation benchmark provides LLMs with a highly challenging and distinctive task that is crucial to an effective AI assistant.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
