COLUMBUS：通过多项选择画谜评估认知水平思维
COLUMBUS: Evaluating COgnitive Lateral Understanding through Multiple-choice reBUSes
AI总结：
提出视觉横向思维多项选择问答任务及三步分类法，构建含千余画谜的COLUMBUS基准，发现视觉语言模型与人类存在显著差距且难以自主生成恰当抽象表示。
摘要：
虽然视觉问答（VQA）基准推动了推理技术的发展，但它们侧重于纵向思维。有效的问题解决还需要横向思维，这在人工智能中仍未得到充分研究，也未被用于测试视觉感知系统。为了弥补这一差距，我们将视觉横向思维表述为多项选择问答任务，并描述了一种三步分类法驱动的方法来实例化任务示例。然后，我们开发了 COLUMBUS，这是一个合成基准，应用该任务流程创建基于公开可用复合词和常见短语集合的文本和图标画谜 QA 集。COLUMBUS 包含超过 1,000 个谜题，每个谜题有四个候选答案。虽然最先进的视觉语言模型（VLMs）取得了不错的性能，但我们的评估表明人类与模型之间存在巨大差距。VLMs 受益于人工策划的描述，但难以在正确的抽象层次上自行生成此类表示。
英文摘要：
While visual question-answering (VQA) benchmarks have catalyzed the development of reasoning techniques, they have focused on vertical thinking. Effective problem-solving also necessitates lateral thinking, which remains understudied in AI and has not been used to test visual perception systems. To bridge this gap, we formulate visual lateral thinking as a multiple-choice question-answering task and describe a three-step taxonomy-driven methodology for instantiating task examples. Then, we develop COLUMBUS, a synthetic benchmark that applies the task pipeline to create QA sets with text and icon rebus puzzles based on publicly available collections of compounds and common phrases. COLUMBUS comprises over 1,000 puzzles, each with four answer candidates. While the SotA vision-language models (VLMs) achieve decent performance, our evaluation demonstrates a substantial gap between humans and models. VLMs benefit from human-curated descriptions but struggle to self-generate such representations at the right level of abstraction.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
