发表机构
Roblox Corporation; Berkeley AI Research Lab & Department of Industrial Engineering and Operations Research, University of California, Berkeley; Computer Science Department, Stanford University; Division of Biostatistics, University of California, Berkeley（罗布乐思公司; 加州大学伯克利分校伯克利人工智能研究实验室及工业工程与运筹学系; 斯坦福大学计算机科学系; 加州大学伯克利分校生物统计学系）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
研究基于视觉语言模型的3D缺陷检测管道评估，引入3D-DefectBench基准框架，通过平衡因子设计改变多个管道因素进行实验，发现模型选择影响最大，各因素相互作用，还得出一些协议表现及评判与人工标注对比情况等结论。
AI 中文摘要
自动化评估对于扩展生成式3D系统至关重要，因为详尽的人工审查成本高且速度慢。然而，自动评判的可靠性取决于整个评估管道，不仅包括基础视觉语言模型（VLM），还包括资产渲染方式、提供的视觉证据、任务指定方式以及人工参考标签的构建方式。我们引入了3D-DefectBench，这是一个用于基于VLM的3D缺陷检测管道系统分析的基准和框架。它通过跨越几何、纹理和提示遵循的九个细粒度二元缺陷来补充整体评分和成对偏好，为生成器开发和评判评估提供可操作的诊断。使用平衡因子设计，我们在84种推理设计和大约320万个评分缺陷决策中改变四个管道因素，即VLM、相机协议、视觉输入和提示模式，随后在更广泛的前沿模型集上进行分阶段验证。模型选择是与人工标签一致性的最大决定因素，但其余因素也会影响性能，与模型选择相互作用，并可能改变最佳配置。在评估的设计空间内，紧凑的六视图RGB协议与更密集的多视图设置以及用深度或表面法线增强的输入表现相当，使其成为强大的性价比高的默认选择。在这个标准化管道下，12个VLM评判中最好的仍然落后于训练有素的人工标注员，而当专家共识标签被更嘈杂的银标签取代时，纹理一致性会急剧下降。这些发现表明，自动评判应作为完整管道进行评估，并在人工参考模式下进行校准，而不是仅作为独立模型进行基准测试。我们在Hugging Face上发布了标签、提示、预测和Croissant元数据。
英文摘要
Automated evaluation is essential for scaling generative 3D systems, where exhaustive human review is costly and slow. Yet the reliability of an automated judge depends on the full evaluation pipeline, including the vision-language model (VLM), asset rendering, visual evidence, task specification, and human reference labels. We introduce 3D-DefectBench, a large-scale benchmark for rigorous evaluation-pipeline analysis. It complements holistic ratings and pairwise preferences with nine fine-grained binary defects spanning geometry, texture, and prompt adherence, with optional human severity annotations. Using a balanced factorial design, we vary the VLM, camera protocol, visual input, and prompt schema across 84 inference designs, and validate the resulting conclusions on a broader set of frontier models. Model choice is the dominant source of variation in agreement with human labels, while other pipeline factors also influence agreement, interact with the model, and can alter the best configuration. A compact six-view RGB protocol performs comparably to denser view sets and configurations augmented with depth or normal channels, making it a strong cost-effective default. Under this fixed design, the best of 12 VLMs still trail trained human labelers, and texture agreement drops sharply from expert-agreement to noisier silver labels. Severity annotations further show that binary judges recover most defects humans flag as severe. These results highlight the importance of evaluating automated judges as complete pipelines and calibrating them across human reference regimes.
