合成设计实验用于诊断视觉模型失效
Synthetic Designed Experiments for Diagnosing Vision Model Failure
AI总结：
本文提出SDRS方法，通过方差分析分解高效审计模型的因子敏感性，识别并解决类型I和II失效缺口，通过针对性合成数据提升模型性能。
摘要：
当前计算机视觉的合成数据管道生成图像而不诊断下游模型实际需求。这种开环范式将合成数据视为廉价真实数据，随机采样生成器输出空间，希望覆盖模型的失效模式。我们主张这种根本上误用了合成数据的独特性质：可控的、独立变化的场景因素。基于统计学的实验设计理论（DoE），我们提出合成设计实验用于表征充分性（SDRS）。SDRS将下游模型视为黑盒系统，合成生成器视为实验装置。利用分数因子设计，SDRS通过方差分析分解高效审计模型的因子敏感性轮廓。它将失效分为两种可操作类型：类型I缺口（在欠代表因子水平上的覆盖失效）和类型II缺口（依赖于虚假干扰依赖）。审计后则为每种缺口类型制定针对性的合成数据。我们验证了SDRS在三个实验中：（1）在dSprites上进行受控诊断，其中审计正确识别了两种缺口类型，并针对性数据将准确率从49.9%提升至79.0%；（2）在程序化场景上的密集分割任务中，检测背景复杂性捷径并应用针对性数据将mIoU从0.948提升至0.998；（3）在纠缠检测实验中，显示方差分析审计能识别不完美生成器中的跨因子污染。最后，我们显示每因子不变性惩罚能转移敏感性，识别出表示层面修正的一个开放问题。
英文摘要：
Current synthetic data pipelines for computer vision generate images without diagnosing what the downstream model actually needs. This open-loop paradigm treats synthetic data as cheap real data, randomly sampling the generator's output space and hoping to cover the model's failure modes. We argue this fundamentally misuses synthetic data's unique property: the controllable, independent variation of scene factors.Drawing on the statistical theory of Design of Experiments (DoE), we propose Synthetic Designed Experiments for Representational Sufficiency (SDRS). SDRS treats the downstream model as a black-box system and the synthetic generator as an experimental apparatus. Using fractional factorial designs, SDRS efficiently audits a model's factor-sensitivity profile via ANOVA decomposition. It classifies failures into two actionable types: Type I gaps (coverage failures on underrepresented factor levels) and Type II gaps (reliance on spurious nuisance dependencies). The audit then prescribes targeted synthetic data to address each gap type. We validate SDRS on three experiments: (1) a controlled diagnostic on dSprites with planted biases, where the audit correctly identifies both gap types and targeted data improves accuracy from 49.9% to 79.0%; (2) a dense segmentation task on procedural scenes, where detecting background-complexity shortcuts and applying targeted data improves mIoU from 0.948 to 0.998; and (3) an entanglement detection experiment showing that the ANOVA audit identifies cross-factor contamination in imperfect generators. Finally, we show that per-factor invariance penalties can transfer sensitivity between factors, identifying an open problem for representation-level correction.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
