Hob-VL：一个用于视觉基础布尔推理的基准
Hob-VL: A Benchmark for Visually Grounded Boolean Reasoning
浏览论文内容
中文总结 AI 辅助
Hob-VL是一个视觉基础布尔推理基准，包含6,000个是/否问题和1,000个对象识别任务，测试模型在误导性线索和嵌套逻辑下的推理能力，结果显示现有模型准确率接近随机水平。
中文摘要 AI 辅助
可靠的视觉推理需要组合多个视觉观察，并对逻辑等价的问题返回一致的答案。我们引入了Hob-VL，一个用于视觉基础布尔推理的基准。Hob-VL包含两个任务：（1）评估图像中是否成立某个布尔规则，（2）识别满足布尔描述的（唯一）对象。Hob-VL包含6,000个人工验证的平衡是/否问题，每个问题由十个视觉陈述的布尔组合定义，覆盖1,000个生成场景和46张多样化的带标签照片，以及基于相同照片的1,000个对象识别问题。我们的问题族被刻意构造，通过误导性的局部线索和嵌套逻辑操作来挑战推理，并包含符号化和结构化的自然语言表述。在八种禁用或最小化思考的模型配置中，布尔准确率从48.52%到50.57%，而识别准确率最多达到43.0%。一种启用思考的GLM配置实现了不均衡的提升，但仍保留大量错误和不一致性。Hob-VL通过可执行的参考答案和匹配评估暴露了这些失败。
英文摘要
Reliable visual reasoning requires composing multiple visual observations and returning consistent answers to logically equivalent questions. We introduce Hob-VL, a benchmark for visually grounded Boolean reasoning. Hob-VL comprises two tasks: (1) evaluating whether a Boolean rule holds in an image, and (2) identifying the (unique) object satisfying a Boolean description. Hob-VL contains 6,000 human-verified balanced Yes/No questions, each defined by a Boolean combination of ten visual statements, across 1,000 generated scenes and 46 diverse labeled photographs, along with 1,000 object-identification questions over the same photographs. Our question families are deliberately constructed to challenge reasoning through misleading local cues and nested logical operations, and include symbolic and structured natural-language presentations. Across eight model configurations with thinking disabled or minimized, Boolean accuracy ranges from 48.52% to 50.57%, while the identification accuracy reaches at most 43.0%. A thinking-enabled GLM configuration achieves uneven gains while retaining substantial errors and inconsistencies. Hob-VL exposes these failures through executable reference answers and matched evaluations.
发表机构
- Georgia Institute of Technology（佐治亚理工学院）
机构由 AI 辅助整理，请以论文原文为准。
