arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2310.08773cs.AIcs.CE
收藏

考察ChatGPT在科学与工程问题求解中的潜力与隐患

Examining the Potential and Pitfalls of ChatGPT in Science and Engineering Problem-Solving

Stanford University(斯坦福大学);Auburn University(奥本大学)

Karen D. Wang, Eric Burkholder, Carl Wieman, Shima Salehi, Nick Haber

AI总结:

本研究通过让ChatGPT求解40道大学工程物理题,发现其在良定义问题上准确率为62.5%,但在欠定义问题上仅8.3%,并归纳出模型失效的三种模式,为STEM教育中利用大语言模型提供了启示。

摘要:

本研究探讨了OpenAI的ChatGPT在求解不同类型物理问题方面的能力。研究使用ChatGPT(搭载GPT-4)求解了一门大学水平工程物理课程中的共40道题目。这些问题既包括所有求解所需数据均已给出的良定义问题,也包括并非所有必要数据都已给出的欠定义现实问题。我们的研究结果表明,ChatGPT能够成功求解62.5%的良定义问题,但在欠定义问题上的准确率降至8.3%。对模型错误解的分析揭示了三种不同的失效模式:1)未能构建准确的物理世界模型;2)未能对缺失数据做出合理假设;3)计算错误。本研究为如何利用大语言模型增强的教学材料来提升STEM教育提供了启示。这些见解也为关于人工智能优势与局限的更广泛讨论做出了贡献,既服务于旨在利用该技术的教育工作者,也服务于研究人类与人工智能协作框架以进行问题求解和决策的研究人员。

英文摘要:

The study explores the capabilities of OpenAI's ChatGPT in solving different types of physics problems. ChatGPT (with GPT-4) was queried to solve a total of 40 problems from a college-level engineering physics course. These problems ranged from well-specified problems, where all data required for solving the problem was provided, to under-specified, real-world problems where not all necessary data were given. Our findings show that ChatGPT could successfully solve 62.5% of the well-specified problems, but its accuracy drops to 8.3% for under-specified problems. Analysis of the model's incorrect solutions revealed three distinct failure modes: 1) failure to construct accurate models of the physical world, 2) failure to make reasonable assumptions about missing data, and 3) calculation errors. The study offers implications for how to leverage LLM-augmented instructional materials to enhance STEM education. The insights also contribute to the broader discourse on AI's strengths and limitations, serving both educators aiming to leverage the technology and researchers investigating human-AI collaboration frameworks for problem-solving and decision-making.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑