Weak-eval-Strong：利用情境谜题评估并激发大语言模型的横向思维
Weak-eval-Strong: Evaluating and Eliciting Lateral Thinking of LLMs with Situation Puzzles
AI总结：
针对LLMs横向思维难评估、缺数据的问题，研究者提出含975道分级情境谜题的SPLAT基准，用多轮玩家-裁判框架降低对强评估模型的依赖，验证了其与人类判断高一致性，且能提升其他横向思维基准的模型性能。
摘要：
尽管自然语言处理（NLP）的进展已大幅提升了大语言模型（LLMs）在需要纵向思维的任务上的表现，但由于创造性思维过程的评估难度大且相关数据稀缺，LLMs的横向思维能力仍未得到充分探索，且难以衡量。为应对这些挑战，我们提出了SPLAT——一个利用情境谜题评估并激发LLMs横向思维的基准（Situation Puzzles to evaluate and elicit LAteral Thinking）。该基准包含975道分三个难度等级的分级情境谜题，采用了一种全新的多轮“玩家-裁判”框架，替代了传统的、通常需要更强评估模型的基于模型的评估方式。该框架模拟一场互动游戏：作为玩家的模型向作为裁判的评估模型提问，围绕一个不完整的故事进行询问，以推断出完整情境。裁判会依据详细的参考情境作答，或评估玩家的预测是否与参考情境一致。这种方法降低了对更强评估模型的依赖，使得对前沿LLMs的评估成为可能。实验表明，像WizardLM-2这样的强评估模型，在中间问答准确率和最终情境准确率上都与人类判断高度接近，一致性超过80%——这一水平与人类之间的一致性程度相近。此外，将我们基准中的数据和推理过程应用到其他横向思维相关基准（例如RiddleSense和BrainTeaser）上，能够带来性能提升。这表明我们的基准能够有效评估并激发LLMs的横向思维能力。代码可在以下链接获取：https://github.com/chenqi008/LateralThinking。
英文摘要：
While advancements in NLP have significantly improved the performance of Large Language Models (LLMs) on tasks requiring vertical thinking, their lateral thinking capabilities remain under-explored and challenging to measure due to the complexity of assessing creative thought processes and the scarcity of relevant data. To address these challenges, we introduce SPLAT, a benchmark leveraging Situation Puzzles to evaluate and elicit LAteral Thinking of LLMs. This benchmark, containing 975 graded situation puzzles across three difficulty levels, employs a new multi-turn player-judge framework instead of the traditional model-based evaluation, which often necessitates a stronger evaluation model. This framework simulates an interactive game where the model (player) asks the evaluation model (judge) questions about an incomplete story to infer the full scenario. The judge answers based on a detailed reference scenario or evaluates if the player's predictions align with the reference one. This approach lessens dependence on more robust evaluation models, enabling the assessment of state-of-the-art LLMs. The experiments demonstrate that a robust evaluation model, such as WizardLM-2, closely matches human judgements in both intermediate question-answering and final scenario accuracy, achieving over 80% agreement-similar to the agreement levels among humans. Furthermore, applying data and reasoning processes from our benchmark to other lateral thinking-related benchmarks, e.g., RiddleSense and BrainTeaser, leads to performance enhancements. This suggests that our benchmark effectively evaluates and elicits the lateral thinking abilities of LLMs. Code is available at: https://github.com/chenqi008/LateralThinking.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
