人工智能能否掌握施工管理（CM）？在CM认证考试中对最先进大语言模型的基准测试
Can AI Master Construction Management (CM)? Benchmarking State-of-the-Art Large Language Models on CM Certification Exams
AI总结：
本研究构建CMExamSet基准数据集，通过CM认证考试零样本评估最先进LLMs，发现GPT-4o和Claude 3.7超人类通过阈值但在图表题和多步推理上有局限，为CM领域LLM应用及优化提供参考。
摘要：
施工管理（CM）项目日益复杂，加上严格的监管要求和劳动力短缺等挑战，需要专门的分析工具来简化项目工作流程并提高绩效。尽管大语言模型（LLMs）在通用推理任务中表现出卓越性能，但它们在应对CM特定挑战（如精确的定量分析和监管解读）方面的有效性仍未得到充分探索。为填补这一空白，本研究引入CMExamSet，这是一个全面的基准数据集，包含来自四个国家认可的CM认证考试的689道真实多项选择题。我们的零样本评估涵盖整体准确率、主题领域（如施工安全）、推理复杂度（单步和多步）以及问题格式（纯文本、参考图表和参考表格）。结果显示，GPT-4o和Claude 3.7超过了典型的人类通过阈值（70%），平均准确率分别为82%和83%。此外，这两个模型在单步任务上表现更好，准确率分别为85.7%（GPT-4o）和86.7%（Claude 3.7）。多步任务更具挑战性，性能分别降至76.5%和77.6%。此外，这两个LLM在参考图表的问题上表现出显著局限性，准确率降至约40%。我们的错误模式分析进一步揭示，概念误解是最常见的（分别为44.4%和47.9%），强调了增强领域特定推理模型的必要性。这些发现凸显了LLMs作为CM中宝贵补充分析工具的潜力，同时也强调了在复杂决策中需要领域特定的改进和持续的人类监督。
英文摘要：
The growing complexity of construction management (CM) projects, coupled with challenges such as strict regulatory requirements and labor shortages, requires specialized analytical tools that streamline project workflow and enhance performance. Although large language models (LLMs) have demonstrated exceptional performance in general reasoning tasks, their effectiveness in tackling CM-specific challenges, such as precise quantitative analysis and regulatory interpretation, remains inadequately explored. To bridge this gap, this study introduces CMExamSet, a comprehensive benchmarking dataset comprising 689 authentic multiple-choice questions sourced from four nationally accredited CM certification exams. Our zero-shot evaluation assesses overall accuracy, subject areas (e.g., construction safety), reasoning complexity (single-step and multi-step), and question formats (text-only, figure-referenced, and table-referenced). The results indicate that GPT-4o and Claude 3.7 surpass typical human pass thresholds (70%), with average accuracies of 82% and 83%, respectively. Additionally, both models performed better on single-step tasks, with accuracies of 85.7% (GPT-4o) and 86.7% (Claude 3.7). Multi-step tasks were more challenging, reducing performance to 76.5% and 77.6%, respectively. Furthermore, both LLMs show significant limitations on figure-referenced questions, with accuracies dropping to approximately 40%. Our error pattern analysis further reveals that conceptual misunderstandings are the most common (44.4% and 47.9%), underscoring the need for enhanced domain-specific reasoning models. These findings underscore the potential of LLMs as valuable supplementary analytical tools in CM, while highlighting the need for domain-specific refinements and sustained human oversight in complex decision making.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
