ScratchEval：一种用于块式编程中LLM的多模态评估框架
ScratchEval : A Multimodal Evaluation Framework for LLMs in Block-Based Programming
AI总结：
ScratchEval是首个用于评估LLM在Scratch块式编程中修复能力的可执行基准，通过多层协议测量功能正确性、修复质量和解释质量，支持领域微调和泛化研究。
摘要：
大型语言模型在文本编程任务上表现优异，但它们在块式语言如Scratch上仍然不可靠。Scratch程序表现出深度嵌套、非线性结构，多个精灵之间的事件驱动并发，以及代码与多媒体资产之间紧密耦合的特性，这些特性与文本代码有根本差异。因此，LLM常常误解Scratch语义，生成大量侵入性修改，这些修改在语法上有效但语义上错误。我们介绍了ScratchEval，这是首个可执行的基准，用于评估LLM在Scratch程序上的修复能力，涵盖程序理解、调试、分析和修复。该基准包含100个经过精心挑选的Scratch项目，这些项目从公共仓库中选取，具有结构和语义复杂性。每个项目都配有可执行的测试套件、带有相应修复的错误描述、块级编辑约束定义最小的语义正确修复，以及必需的多媒体资产。该基准通过结合自动化项目挖掘与专家验证触发-结果语义和代表性错误模式的人工在循环流程构建，重点放在事件顺序、并发和状态管理上。为了实现严格且可重复的评估，我们提出了一种三层可执行协议，通过虚拟机级别执行测量功能正确性，使用块级编辑距离和行为轨迹比较测量修复质量，并通过结构化评分表评估解释质量。使用ScratchEval，我们研究了领域特定的微调、训练数据有效性以及模型对未见过的错误类型的泛化能力。ScratchEval为评估和微调块式编程任务上的LLM提供了可重复的基础。
英文摘要：
LLMs have achieved strong performance on text-based programming tasks, yet they remain unreliable for block-based languages such as Scratch. Scratch programs exhibit deeply nested, non-linear structures, event-driven concurrency across multiple sprites, and tight coupling between code and multimedia assets, properties that differ fundamentally from textual code. As a result, LLMs often misinterpret Scratch semantics and generate large, invasive edits that are syntactically valid but semantically incorrect when repairing buggy programs. We introduce ScratchEval, the first executable benchmark designed to evaluate LLM-based repair for Scratch programs, covering program understanding, debugging, analysis, and repair. The benchmark contains 100 curated Scratch projects from the public repository, selected for structural and semantic complexity. Each project is paired with executable test suites, bug descriptions with corresponding fixes, block-level edit constraints defining minimal semantically correct repairs, and required multimedia assets. The benchmark is constructed through a human-in-the-loop pipeline combining automated project mining with expert validation of trigger-outcome semantics and representative bug patterns, with emphasis on event ordering, concurrency, and state management. To enable rigorous and reproducible evaluation, we propose a three-layer executable protocol measuring functional correctness via VM-level execution, repair quality using block-level edit distance and behavioral trajectory comparisons, and explanation quality via structured rubrics assessing alignment between model reasoning and generated patches. Using ScratchEval, we study domain-specific fine-tuning, training data effectiveness, and model generalization to unseen bug types. ScratchEval provides a reproducible foundation for evaluating and post-training LLMs on block-based programming tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
