Introduction: As system dynamics (SD) embraces automation, AI offers efficiency but risks bias from missing data and flawed models. Models that omit multiple perspectives and data threaten model quality, whether created by humans or with the assistance of AI. To reduce uncertainty about how well AI can build SD models, we introduce two metrics for evaluation of AI-generated causal maps: technical correctness (causal translation) and adherence to instructions (conformance).
Approach: We developed an open source project called sd-ai to provide a basis for collaboration in the SD community, aiming to fully harness the potential of AI based tools like ChatGPT for dynamic modeling. Additionally, we created an evaluation theory along with a comprehensive suite of tests designed to evaluate any such tools developed within the sd-ai ecosystem.
Results: We tested 11 different LLMs on their ability to do causal translation as well as conform to user instruction. gpt-4.5-preview was the top performer, scoring 92.9% overall, excelling in both tasks. o1 scored 100% in causal translation. gpt-4o identified all causal links but struggled with positive polarity in decreasing terms. While gpt-4.5-preview and o1 are most accurate, gpt-4o is the cheapest.
Discussion: Causal translation and conformance tests applied to the sd-ai engine reveal significant variations across lLLMs, underscoring the need for continued evaluation to ensure responsible development of AI tools for dynamic modeling. To address this, an open collaboration among tool developers, modelers, and stakeholders is launched to standardize measures for evaluating the capacity of AI tools to improve the modeling process.
Generating Findings for Jaw Cysts in Dental Panoramic Radiographs Using a GPT-Based VLM: A Preliminary Study on Building a Two-Stage Self-Correction Loop with Structured Output (SLSO) Framework
Gifu University(岐阜大学);EyeTech Co., Ltd(EyeTech有限公司);Asahi University(朝日大学);Shiga University(滋贺大学);The University of Osaka Dental Hospital(大阪大学齿科医院)
How Well Do Large Language Models Serve as End-to-End Secure Code Agents for Python?
School of Computer Science and Engineering, Beihang University(北京航空航天大学计算机科学与工程学院);State Key Laboratory of Software Development Environment(软件开发环境国家重点实验室);Zhongguancun Laboratory(中关村实验室);Tsinghua University(清华大学)
Large Language Models (GPT) for automating feedback on programming assignments
Institute of Information Technology, Warsaw University of Life Sciences(华沙生命科学大学信息技术研究所);Penn Center for Learning Analytics, University of Pennsylvania(宾夕法尼亚大学宾夕法尼亚学习分析中心)
An evaluation of GPT models for phenotype concept recognition
Rare Care Centre(罕见病护理中心);Perth Children's Hospital(珀斯儿童医院);Telethon Kids Institute(Telethon儿童研究所);School of Electrical Engineering, Computing and Mathematical Sciences, Curtin University(科廷大学电气工程、计算与数学科学学院);SingHealth Duke-NUS Institute of Precision Medicine(新加坡保健服务集团杜克-新加坡国立大学精准医学研究所);Lawrence Berkeley National Laboratory(劳伦斯伯克利国家实验室);Western Australian Register of Developmental Anomalies(西澳大利亚发育异常登记处);King Edward Memorial Hospital(爱德华国王纪念医院);Faculty of Health and Medical Sciences, University of Western Australia(西澳大利亚大学健康与医学科学学院);University of Colorado Anschutz Medical Campus(科罗拉多大学安舒茨医学校区);The Jackson Laboratory for Genomic Medicine(杰克逊基因组医学实验室);Institute for Systems Genomics, University of Connecticut(康涅狄格大学系统基因组学研究所)
Vrije University of Amsterdam University of Amsterdam Centrum Wiskunde \& Informatic;University of Amsterdam National Institute of Informatics Centrum Wiskunde \& Informatic;Centrum Wiskunde \& Informatic Leiden University University College London;Vrije University of Amsterdam;Centrum Wiskunde \& Informatica Technische Universiteit Delft Guangzhou University;Vrije University of Amsterdam Centrum Wiskunde \& Informatic
How Well Does GPT-4V(ision) Adapt to Distribution Shifts? A Preliminary Investigation
Microsoft Research Asia(微软亚洲研究院);Westlake University(西湖大学);School of Software, Shandong University(山东大学软件学院);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);Australian National University(澳大利亚国立大学);The University of New South Wales(新南威尔士大学);Data61, CSIRO(联邦科学与工业研究组织Data61);Macquarie University(麦考瑞大学);The University of Sydney(悉尼大学);Carnegie Mellon University(卡内基梅隆大学)
From GPT-3 to GPT-5: Mapping their capabilities, scope, limitations, and consequences
University of Bergen(卑尔根大学);Norwegian University of Life Sciences(挪威生命科学大学);Sohar University(苏哈尔大学);Norwegian University of Science and Technology(挪威科技大学);Kristiania University of Applied Sciences(克里斯蒂安尼亚应用科学大学)