arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2411.08320cs.AI
收藏

建筑安全中的负责任AI:大型语言模型与提示工程的系统评估

Responsible AI in Construction Safety: Systematic Evaluation of Large Language Models and Prompt Engineering

Farouq Sammour, Jia Xu, Xi Wang, Mo Hu, Zhenyu Zhang

AI总结:

本研究评估了GPT-3.5和GPT-4o在BCSP标准化安全考试中的表现,发现两者均超越基准,并揭示了提示工程对准确性的显著影响,为AI在建筑安全中的负责任整合提供了实证依据。

摘要:

建筑业仍然是最危险的行业之一。人工智能的最新进展,特别是大型语言模型(LLMs),为提升工作场所安全提供了有前景的机遇。然而,负责任地整合LLMs需要进行系统评估,因为在不了解其能力和局限性的情况下部署它们,会面临生成不准确信息、引发错误信心以及危及工人安全的风险。本研究评估了两种广泛使用的LLMs(GPT-3.5和GPT-4o)在认证安全专业人员委员会(BCSP)管理的三项标准化考试中的表现。研究使用了涵盖七个安全知识领域的385道题目,分析了模型的准确性、一致性和可靠性。结果表明,两个模型均持续超过BCSP基准,其中GPT-4o的准确率达到84.6%,GPT-3.5达到73.8%。两个模型在安全管理系统和危险识别与控制方面表现出优势,但在科学、数学、应急响应和消防预防方面存在弱点。错误分析识别出影响LLM性能的四个主要局限:知识缺乏、推理缺陷、记忆问题和计算错误。本研究还强调了提示工程策略的影响,GPT-3.5的准确率变化达到13.5%,GPT-4o达到7.9%。然而,没有任何单一的提示配置被证明是普遍有效的。本研究通过以下三个方面推进了知识发展:识别了LLMs可以支持安全实践的领域以及仍需人工监督的领域,提供了通过提示工程改进LLM实施的实用见解,并为未来的研究和开发提供了基于证据的方向。这些贡献支持了人工智能在建筑安全管理中的负责任整合,以实现零伤害的目标。

英文摘要:

Construction remains one of the most hazardous sectors. Recent advancements in AI, particularly Large Language Models (LLMs), offer promising opportunities for enhancing workplace safety. However, responsible integration of LLMs requires systematic evaluation, as deploying them without understanding their capabilities and limitations risks generating inaccurate information, fostering misplaced confidence, and compromising worker safety. This study evaluates the performance of two widely used LLMs, GPT-3.5 and GPT-4o, across three standardized exams administered by the Board of Certified Safety Professionals (BCSP). Using 385 questions spanning seven safety knowledge areas, the study analyzes the models' accuracy, consistency, and reliability. Results show that both models consistently exceed the BCSP benchmark, with GPT-4o achieving an accuracy rate of 84.6% and GPT-3.5 reaching 73.8%. Both models demonstrate strengths in safety management systems and hazard identification and control, but exhibit weaknesses in science, mathematics, emergency response, and fire prevention. An error analysis identifies four primary limitations affecting LLM performance: lack of knowledge, reasoning flaws, memory issues, and calculation errors. Our study also highlights the impact of prompt engineering strategies, with variations in accuracy reaching 13.5% for GPT-3.5 and 7.9% for GPT-4o. However, no single prompt configuration proves universally effective. This research advances knowledge in three ways: by identifying areas where LLMs can support safety practices and where human oversight remains essential, by offering practical insights into improving LLM implementation through prompt engineering, and by providing evidence-based direction for future research and development. These contributions support the responsible integration of AI in construction safety management toward achieving zero injuries.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑