视觉下的推理：理解视觉语言模型在验证码（CAPTCHA）中的视觉空间认知
Reasoning under Vision: Understanding Visual-Spatial Cognition in Vision-Language Models for CAPTCHA
AI总结：
本研究表明分步推理对视觉语言模型解决验证码至关重要，提出首个带推理的验证码基准CAPTCHA-X及通用智能体VLM框架，大幅提升准确率至83.9%，为视觉空间推理研究提供新方向。
摘要：
验证码（CAPTCHA）最初设计用于区分人类与机器人，现已演变为评估视觉语言模型空间推理能力的真实世界基准。在本研究中，我们首先表明，分步推理对于视觉语言模型（VLMs）解决验证码至关重要——验证码代表高难度空间推理任务，而当前商业视觉语言模型在这类推理中仍存在困难。具体而言，我们观察到大多数商业VLMs（如Gemini、Claude、GPT等）无法有效解决验证码，因此准确率较低（约21.9%）。然而，我们的研究结果显示，要求模型在生成最终坐标前执行分步推理可显著提高其解决准确率，这凸显了差距的严重性。为系统研究该问题，我们引入CAPTCHA-X，这是首个带推理的真实世界验证码基准，涵盖七类验证码（如五子棋、hCaptcha等），包含分步动作解决方案和定位标注。我们进一步定义了五个面向推理的指标，能够全面评估模型的推理能力。为验证推理的有效性，我们还提出一个通用的基于智能体VLM的框架，该框架整合了模型固有的推理能力。我们的方法在五类高难度验证码上取得了最先进的性能，平均解决准确率达83.9%，大幅超越现有基线。这些结果揭示了当前模型的局限性，并强调了推理在未来推进视觉空间挑战中的重要性。
英文摘要：
CAPTCHA, originally designed to distinguish humans from robots, has evolved into a real-world benchmark for assessing the spatial reasoning capabilities of vision-language models. In this work, we first show that step-by-step reasoning is crucial for vision-language models (VLMs) to solve CAPTCHAs, which represent high-difficulty spatial reasoning tasks, and that current commercial vision-language models still struggle with such reasoning. In particular, we observe that most commercial VLMs (e.g., Gemini, Claude, GPT, etc.) fail to effectively solve CAPTCHAs and thus achieve low accuracy (around 21.9 percent). However, our findings indicate that requiring the model to perform step-by-step reasoning before generating the final coordinates can significantly enhance its solving accuracy, underscoring the severity of the gap. To systematically study this issue, we introduce CAPTCHA-X, the first real-world CAPTCHA benchmark with reasoning, covering seven categories of CAPTCHAs (such as Gobang, hCaptcha, etc.) with step-by-step action solutions and grounding annotations. We further define five reasoning-oriented metrics that enable a comprehensive evaluation of models reasoning capabilities. To validate the effectiveness of reasoning, we also propose a general agentic VLM-based framework that incorporates the models inherent reasoning abilities. Our method achieves state-of-the-art performance across five high-difficulty CAPTCHA types, with an average solving accuracy of 83.9 percent, substantially surpassing existing baselines. These results reveal the limitations of current models and highlight the importance of reasoning in advancing visual-spatial challenges in the future.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
