arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.18520cs.CVcs.AI
收藏

Sketch2Feedback: 用于学生STEM图表的基于语法规则的反馈框架

Sketch2Feedback: Grammar-in-the-Loop Framework for Rubric-Aligned Feedback on Student STEM Diagrams

IEEE

Aayam Bansal

AI总结:

Sketch2Feedback通过结合语法规则和视觉语言模型,提高了学生STEM图表反馈的准确性与可靠性,展示了语法规则与端到端方法的互补性。

摘要:

在STEM教育中,为学生绘制的图表提供及时且符合评分标准的反馈是一个持续的挑战。尽管大型多模态模型(LMMs)能够联合解析图像并生成解释,但其倾向于产生幻觉,这削弱了在课堂上的信任度。我们提出了Sketch2Feedback,一个基于语法规则的框架,将问题分解为四个阶段——混合感知、符号图构建、约束检查和受约束的视觉语言模型反馈,使得语言模型仅对经过上游规则引擎验证的违规情况进行描述。我们在两个合成微基准测试中进行了评估,FBD-10(自由体图)和Circuit-10(电路图),每个测试集包含500张图像,涵盖标准和困难的噪声增强层级,将我们的流程与端到端LMMs(LLaVA-1.5-7B、Qwen2-VL-7B)、一个仅基于视觉的检测器、一个YOLOv8-nano学习的检测器以及一个集成的Oracle进行比较。在n=100个测试样本每基准,95%的bootstrap置信区间下,结果混合且具有指导意义:Qwen2-VL-7B在FBD上实现了最高的微F1(0.570)和电路(0.528),但幻觉率极高(0.78,0.98)。一个选择每个样本最佳预测的集成Oracle在FBD上达到F1=0.556,幻觉率为0.320,表明语法规则和端到端方法之间存在可利用的互补性。在置信度阈值tau=0.7时,电路幻觉率从0.970降至0.880,没有F1损失。困难噪声增强揭示了领域依赖的鲁棒性:FBD检测具有鲁棒性,而电路检测则急剧下降。一个LLM作为评判者的评估确认,语法规则流程产生的电路反馈(4.85/5)比端到端LMM(3.11/5)更具可操作性。我们发布了所有代码、数据集和评估脚本。

英文摘要:

Providing timely, rubric-aligned feedback on student-drawn diagrams is a persistent challenge in STEM education. While large multimodal models (LMMs) can jointly parse images and generate explanations, their tendency to hallucinate undermines trust in classroom deployments. We present Sketch2Feedback, a grammar-in-the-loop framework that decomposes the problem into four stages -- hybrid perception, symbolic graph construction, constraint checking, and constrained VLM feedback -- so that the language model verbalizes only violations verified by an upstream rule engine. We evaluate on two synthetic micro-benchmarks, FBD-10 (free-body diagrams) and Circuit-10 (circuit schematics), each with 500 images spanning standard and hard noise augmentation tiers, comparing our pipeline against end-to-end LMMs (LLaVA-1.5-7B, Qwen2-VL-7B), a vision-only detector, a YOLOv8-nano learned detector, and an ensemble oracle. On n=100 test samples per benchmark with 95% bootstrap CIs, results are mixed and instructive: Qwen2-VL-7B achieves the highest micro-F1 on both FBDs (0.570) and circuits (0.528), but with extreme hallucination rates (0.78, 0.98). An ensemble oracle that selects the best prediction per sample reaches F1=0.556 with hallucination 0.320 on FBDs, demonstrating exploitable complementarity between grammar and end-to-end approaches. Confidence thresholding at tau=0.7 reduces circuit hallucination from 0.970 to 0.880 with no F1 loss. Hard noise augmentation reveals domain-dependent robustness: FBD detection is resilient while circuit detection degrades sharply. An LLM-as-judge evaluation confirms that the grammar pipeline produces more actionable circuit feedback (4.85/5) than the end-to-end LMM (3.11/5). We release all code, datasets, and evaluation scripts.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑