像素上的模式：测量多模态代码生成中的模式完成偏差
Pattern over Pixels: Measuring Pattern Completion Bias in Multimodal Code Generation
浏览论文内容
中文总结 AI 辅助
该研究针对多模态代码生成中存在的模式完成偏差问题，构建了首个视觉模式完成偏差基准，评估发现前沿MLLMs均存在严重偏差，且偏差与视觉显著性密切相关。
中文摘要 AI 辅助
多模态大语言模型（MLLMs）越来越多地被用于将网页截图转换为前端代码，但重复的UI模式可能会使它们产生视觉上不正确但与模式一致的输出。在这项工作中，我们测试了重复的网页模式如何损害MLLM在客观的截图到代码填空任务上的准确性。我们推出了首个用于视觉模式完成偏差的基准，其中重复UI模式中的一个局部元素被扰动，模型必须从截图和HTML上下文中恢复被掩码的宽度或字体大小值。我们从Design2Code数据集中筛选的30个网页出发，构建了1440张评估截图，涵盖标准和噪声叠加条件下的结构卡片和文本样式模式。我们评估了五个前沿MLLMs，发现所有模型都强烈偏向重复的基线。在卡片宽度扰动上的平均偏差率达到69.78%，在文本字体大小扰动上达到80.22%，而平均准确率仅分别为21.17%和7.89%。Codex-5.3表现最佳，但在卡片上的准确率仍从68.61%降至文本上的13.89%，而Flash-3.0在文本上达到96.11%的偏差。噪声、更细微的扰动以及边界位置进一步提高了偏差率。推理分析进一步表明，更多的推理努力与更低的偏差相关，但定性证据显示，模型可以识别异常元素，却仍会用与模式一致的答案覆盖它。我们的结果确定了多模态代码生成中的一种具体故障模式，并表明其严重程度与视觉显著性密切相关。
英文摘要
Multimodal large language models (MLLMs) are increasingly used to translate webpage screenshots into front-end code, but repeated UI patterns may sway them toward visually incorrect yet pattern-consistent outputs. In this work, we test how repeated webpage patterns hurt MLLM accuracy on an objective screenshot-to-code fill-in-the-blank task. We introduce the first benchmark for visual pattern-completion bias, where one localized element in a repeated UI pattern is perturbed and the model must recover the masked width or font-size value from the screenshot and HTML context. Starting from 30 webpages curated from the Design2Code dataset, we build 1,440 evaluated screenshots spanning structural card and text-style patterns under standard and noise-overlaid conditions. We evaluate five frontier MLLMs and find that all are strongly biased toward the repeated baseline. Mean bias rate reaches 69.78% on card-width perturbations and 80.22% on text font-size perturbations, while mean accuracy is only 21.17% and 7.89%, respectively. Codex-5.3 performs best but still drops from 68.61% accuracy on cards to 13.89% on text, while Flash-3.0 reaches 96.11% bias on text. Noise, subtler perturbations, and boundary positions further increase bias rate. Reasoning analysis further shows that greater reasoning effort correlates with lower bias, yet qualitative evidence reveals that models can identify the anomalous element and still override it with the pattern-consistent answer. Our results identify a concrete failure mode in multimodal code generation and show that its severity is strongly associated with visual saliency
发表机构
- William & Mary（威廉与玛丽学院）
机构由 AI 辅助整理，请以论文原文为准。
