arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2503.15580cs.AI
收藏

AI构建SD模型的能力如何?

How Well Can AI Build SD Models?

William Schoenberg, Davidson Girard, Saras Chung, Ellen O'Neill, Janet Velasquez, Sara Metcalf

AI总结:

本研究针对AI构建系统动力学模型的能力评估问题,提出因果转换与一致性两项指标,开发sd-ai开源项目及测试套件,测试11种LLM发现性能差异显著,倡议社区协作标准化评估以推动负责任开发。

摘要:

引言:随着系统动力学(SD)走向自动化,AI在提升效率的同时,也存在因数据缺失和模型缺陷带来的偏差风险。无论是由人类创建还是在AI辅助下创建的模型,若忽略多维度视角与多源数据,都会危及模型质量。为降低关于AI构建SD模型能力的不确定性,我们提出两项用于评估AI生成的因果图的指标:技术正确性(因果转换)与指令遵循度(一致性)。\n方法:我们开发了名为sd-ai的开源项目,为SD领域的社区协作提供基础,旨在充分发挥ChatGPT等AI工具在动态建模中的潜力。此外,我们构建了一套评估理论与全面的测试套件,用于评估sd-ai生态系统内开发的任何此类工具。\n结果:我们测试了11种不同大语言模型(LLM)的因果转换能力与用户指令遵循能力。gpt-4.5-preview表现最佳,总体得分92.9%,在两项任务中均表现优异。o1在因果转换任务中得分100%。gpt-4o能识别所有因果关联,但在递减项的正极性处理上存在困难。gpt-4.5-preview与o1的准确率最高,而gpt-4o的成本最低。\n讨论:应用于sd-ai引擎的因果转换与一致性测试显示,不同LLM之间存在显著差异,凸显了持续开展评估以确保动态建模AI工具负责任开发的必要性。为此,工具开发者、建模人员与利益相关方已启动开放协作,旨在标准化AI工具提升建模流程能力的评估度量标准。

英文摘要:

Introduction: As system dynamics (SD) embraces automation, AI offers efficiency but risks bias from missing data and flawed models. Models that omit multiple perspectives and data threaten model quality, whether created by humans or with the assistance of AI. To reduce uncertainty about how well AI can build SD models, we introduce two metrics for evaluation of AI-generated causal maps: technical correctness (causal translation) and adherence to instructions (conformance). Approach: We developed an open source project called sd-ai to provide a basis for collaboration in the SD community, aiming to fully harness the potential of AI based tools like ChatGPT for dynamic modeling. Additionally, we created an evaluation theory along with a comprehensive suite of tests designed to evaluate any such tools developed within the sd-ai ecosystem. Results: We tested 11 different LLMs on their ability to do causal translation as well as conform to user instruction. gpt-4.5-preview was the top performer, scoring 92.9% overall, excelling in both tasks. o1 scored 100% in causal translation. gpt-4o identified all causal links but struggled with positive polarity in decreasing terms. While gpt-4.5-preview and o1 are most accurate, gpt-4o is the cheapest. Discussion: Causal translation and conformance tests applied to the sd-ai engine reveal significant variations across lLLMs, underscoring the need for continued evaluation to ensure responsible development of AI tools for dynamic modeling. To address this, an open collaboration among tool developers, modelers, and stakeholders is launched to standardize measures for evaluating the capacity of AI tools to improve the modeling process.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑