BRAINTEASER：大型语言模型的横向思维谜题
BRAINTEASER: Lateral Thinking Puzzles for Large Language Models
AI总结：
针对语言模型缺乏横向思维研究的问题，本文提出BRAINTEASER基准，通过三步流程构建1,100个谜题，实验显示模型与人类性能差距显著，并公开代码数据以推动该领域发展。
摘要：
语言模型的成功激发了自然语言处理社区对需要隐式和复杂推理任务的关注，这些任务依赖于类人的常识机制。虽然这类垂直思维任务相对流行，但横向思维谜题却很少受到关注。为了弥补这一差距，我们设计了BRAINTEASER：一个多项选择问答任务，旨在测试模型展现横向思维和违背默认常识关联的能力。我们设计了一个三步流程来创建第一个横向思维基准，包括数据收集、干扰项生成和对抗性示例生成，最终产生了1,100个具有高质量标注的谜题。为了评估模型横向推理的一致性，我们基于对问题的语义和上下文重构来丰富BRAINTEASER。我们使用最先进的指令和常识语言模型进行的实验揭示了人类与模型性能之间的显著差距，当考虑对抗性格式的一致性时，这一差距进一步扩大。我们公开了所有代码和数据，以激励开发和评估横向思维模型的工作。
英文摘要：
The success of language models has inspired the NLP community to attend to tasks that require implicit and complex reasoning, relying on human-like commonsense mechanisms. While such vertical thinking tasks have been relatively popular, lateral thinking puzzles have received little attention. To bridge this gap, we devise BRAINTEASER: a multiple-choice Question Answering task designed to test the model's ability to exhibit lateral thinking and defy default commonsense associations. We design a three-step procedure for creating the first lateral thinking benchmark, consisting of data collection, distractor generation, and generation of adversarial examples, leading to 1,100 puzzles with high-quality annotations. To assess the consistency of lateral reasoning by models, we enrich BRAINTEASER based on a semantic and contextual reconstruction of its questions. Our experiments with state-of-the-art instruction- and commonsense language models reveal a significant gap between human and model performance, which is further widened when consistency across adversarial formats is considered. We make all of our code and data available to stimulate work on developing and evaluating lateral thinking models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
