设计中的AI评审：基于视觉-语言模型实现人类专家等效的统计视角
AI Judges in Design: Statistical Perspectives on Achieving Human Expert Equivalence With Vision-Language Models
AI总结：
本文提出一个严格的统计框架来判定AI评审打分是否与人类专家等效，案例研究表明，结合推理与多模态上下文学习的视觉-语言模型在设计评估中能达到人类专家水平。
摘要：
早期工程设计（如概念草图）的主观评估传统上依赖于人类专家。然而，专家评估耗时、昂贵，且有时不一致。视觉-语言模型（VLMs）的最新进展为设计评估自动化提供了可能，但确保这些AI“评审”的表现与人类专家相当至关重要。然而，目前尚无评估专家等效性的框架。本文提出了一个严格的统计框架，用于判定AI评审的打分是否与人类专家相匹配。我们将该框架应用于一项案例研究，评估了四种基于VLM的AI评审在关键设计指标（独特性、创造力、实用性和绘图质量）上的表现。这些AI评审采用了多种上下文学习（ICL）技术，包括单模态与多模态提示以及推理时推理。相同的统计框架也被用于评估三名受训新手是否达到专家等效水平。结果显示，表现最好的AI评审结合了基于文本和图像的ICL与推理，在独特性和绘图质量上达到了专家级一致性，并在所有指标上优于或匹敌受训新手。在独特性和创造力的6/6次运行中，以及在绘图质量和实用性的5/6次运行中，其与专家的一致性达到或超过了大多数受训新手。这些发现表明，支持推理的VLM模型可以在设计评估中实现人类专家等效性。这对于在教育与实践领域扩展设计评估规模具有重要意义，并为在其他需要主观内容评估的领域验证AI评审提供了一个通用的统计框架。
英文摘要：
The subjective evaluation of early stage engineering designs, such as conceptual sketches, traditionally relies on human experts. However, expert evaluations are time-consuming, expensive, and sometimes inconsistent. Recent advances in vision-language models (VLMs) offer the potential to automate design assessments, but it is crucial to ensure that these AI ``judges'' perform on par with human experts. However, no existing framework assesses expert equivalence. This paper introduces a rigorous statistical framework to determine whether an AI judge's ratings match those of human experts. We apply this framework in a case study evaluating four VLM-based judges on key design metrics (uniqueness, creativity, usefulness, and drawing quality). These AI judges employ various in-context learning (ICL) techniques, including uni- vs. multimodal prompts and inference-time reasoning. The same statistical framework is used to assess three trained novices for expert-equivalence. Results show that the top-performing AI judge, using text- and image-based ICL with reasoning, achieves expert-level agreement for uniqueness and drawing quality and outperforms or matches trained novices across all metrics. In 6/6 runs for both uniqueness and creativity, and 5/6 runs for both drawing quality and usefulness, its agreement with experts meets or exceeds that of the majority of trained novices. These findings suggest that reasoning-supported VLM models can achieve human-expert equivalence in design evaluation. This has implications for scaling design evaluation in education and practice, and provides a general statistical framework for validating AI judges in other domains requiring subjective content evaluation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
