DrawingBench:通过基于鼠标绘制的任务评估大语言模型的空间推理和UI交互能力
DrawingBench: Evaluating Spatial Reasoning and UI Interaction Capabilities of Large Language Models through Mouse-Based Drawing Tasks
AI总结:
DrawingBench通过基于鼠标绘制的任务评估大语言模型的空间推理和UI交互能力,揭示了模型在复杂场景中的表现及外部监督的重要性。
摘要:
随着代理AI系统越来越多地自主运行,通过可验证的评估建立信任变得至关重要。然而,现有基准缺乏评估代理行为是否可靠所需的透明度和可审计性。我们提出了DrawingBench,一种用于通过需要生成低级GUI操作序列的空间推理任务来评估代理LLM可信度的验证框架。与不透明的评估不同,DrawingBench提供透明、基于规则的评估:8个客观标准使评分可重复,而操作级检查允许利益相关者审计代理行为。我们的框架包括250个跨20个类别和4个难度级别的多样化提示、确定性评估指标,以及通过多轮反馈实现的外部监督机制,使人类能够控制代理的改进。在对四个最先进的LLM(Claude-4 Sonnet、GPT-4.1、GPT-4.1-mini、Gemini-2.5 Flash)进行1000次测试的评估中,我们确定了能力和限制:模型在结构化外部反馈驱动下实现了92.8%的完美表现(平均+3.2%,复杂场景最高+32.8%),但在工具状态管理和长周期规划中出现了系统性错误模式。值得注意的是,规范清晰度比任务复杂性更重要——当给定明确、可验证的准则时,模型实现了100%的完美表现。这些发现表明,透明的评估框架可以建立对代理系统的信任,外部监督比自我纠正更可靠地指导代理行为。我们的开源框架为可信代理评估提供了模板。代码和数据:https://github.com/hyunjun1121/DrawingBench
英文摘要:
As agentic AI systems increasingly operate autonomously, establishing trust through verifiable evaluation becomes critical. Yet existing benchmarks lack the transparency and auditability needed to assess whether agents behave reliably. We present DrawingBench, a verification framework for evaluating the trustworthiness of agentic LLMs through spatial reasoning tasks that require generating sequences of low-level GUI actions. Unlike opaque evaluations, DrawingBench provides transparent, rule-based assessment: 8 objective criteria enable reproducible scoring, while action-level inspection allows stakeholders to audit agent behavior. Our framework comprises 250 diverse prompts across 20 categories and 4 difficulty levels, deterministic evaluation metrics, and an external oversight mechanism through multi-turn feedback that enables human control over agent refinement. Evaluating four state-of-the-art LLMs (Claude-4 Sonnet, GPT-4.1, GPT-4.1-mini, Gemini-2.5 Flash) across 1,000 tests, we establish both capabilities and limitations: models achieved 92.8% perfect performance with structured external feedback driving significant improvements (average +3.2%, up to +32.8% for complex scenes), but systematic error patterns emerged in tool state management and long-horizon planning. Notably, specification clarity proved more important than task complexity -- models achieved 100% perfect performance when given explicit, verifiable criteria. These findings demonstrate that transparent evaluation frameworks can establish trust in agentic systems, with external oversight proving more reliable than self-correction for guiding agent behavior. Our open-source framework provides a template for trustworthy agent assessment. Code and data: https://github.com/hyunjun1121/DrawingBench
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。