TurtleAI：海龟图形学中视觉编程的多模态模型基准测试
TurtleAI: Benchmarking Multimodal Models for Visual Programming in Turtle Graphics
AI总结：
提出TurtleAI基准，包含823个基于海龟图形学真实任务的视觉编程任务，评估20多个多模态模型发现成功率低于30%，并通过少量种子样本生成合成数据微调Qwen2-VL-72B提升约20%性能。
摘要：
视觉语言模型（VLM）已被探索用于视觉编程，即生成代码以解决视觉任务。然而，大多数先前工作侧重于提高生产力的视觉编程；目前尚不清楚当前VLM在教育导向的视觉编程上表现如何，以及哪些因素限制了它们的性能。为填补这一空白，我们引入了TurtleAI，这是一个包含823个任务的基准，这些任务基于海龟图形学领域的真实视觉编程任务精心策划。解决这些任务需要模型感知几何图案、推理空间关系，并合成能忠实再现几何图案的Python代码。我们评估了20多个VLM，包括GPT-5、GPT-4o和Qwen2-VL-72B，发现它们表现显著困难，大多数成功率低于30%。为解决这些限制，我们提出了一种仅需少量种子样本的数据生成技术。在生成的合成数据上微调Qwen2-VL-72B，在真实任务上取得了约20%的提升。我们的失败分析揭示，GPT-4o在空间推理和精确视觉复制方面存在困难，而微调主要改善了视觉推理与代码实现之间的对齐。
英文摘要：
Vision-language models (VLMs) have been explored for visual programming, where they generate code to solve visual tasks. However, most prior work focuses on visual programming for productivity; it remains unclear how well current VLMs perform on education-oriented visual programming and what factors limit their performance. To bridge this gap, we introduce TurtleAI, a benchmark containing 823 tasks curated based on real-world visual programming tasks in the Turtle Graphics domain. Solving these tasks requires models to perceive geometric patterns, reason about spatial relationships, and synthesize Python code that faithfully reproduces geometric patterns. We evaluate 20+ VLMs, including GPT-5, GPT-4o, and Qwen2-VL-72B, and find that they struggle significantly, with most achieving success rates below 30%. To address these limitations, we propose a data generation technique that requires only a small set of seed samples. Fine-tuning Qwen2-VL-72B on the resulting synthetic data yields an improvement of about 20% on real-world tasks. Our failure analysis reveals that GPT-4o struggles with spatial reasoning and precise visual replication, whereas fine-tuning primarily improves the alignment between visual reasoning and code implementation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
