评估机械工程教育中的大型语言模型：一项聚焦力学的概念理解研究
Assessing Large Language Models in Mechanical Engineering Education: A Study on Mechanics-Focused Conceptual Understanding
AI总结：
该研究用126道力学多选题评估GPT-3.5、GPT-4和Claude-2.1，发现GPT-4总体优于其他模型及人类群体，解释型提示能显著提升表现。
摘要：
本研究是一项开创性工作，旨在调查大型语言模型（LLM）在解决机械工程领域、重点聚焦力学的概念性问题方面的能力。我们的考查使用一份手工编制的试卷，包含126道多项选择题，涵盖力学课程的各个方面，包括流体力学、机械振动、工程静力学与动力学、材料力学、弹性理论以及连续介质力学。研究对三个LLM（ChatGPT（GPT-3.5）、ChatGPT（GPT-4）和Claude（Claude-2.1））进行了评估，并与具有或不具有机械工程背景的工程院系教师和学生进行比较。研究结果表明，在回答各类力学主题问题时，GPT-4的表现优于其他两个LLM及人类群体，但连续介质力学除外。这表明GPT模型在处理符号计算和张量分析方面仍有未来改进潜力。当在直接回答之前提示模型给出解释时，所有LLM的表现均显著提高，凸显了提示工程的关键作用。有趣的是，GPT-3.5在提示涵盖更广泛领域时表现提升，而GPT-4则在提示聚焦特定学科时表现更佳。最后，GPT-4在缓解输入偏差方面展现出显著进步，人类的猜测偏好可作为证据。本研究揭示了LLM作为知识渊博的助手在机械教学和科学研究中的巨大潜力。
英文摘要：
This study is a pioneering endeavor to investigate the capabilities of Large Language Models (LLMs) in addressing conceptual questions within the domain of mechanical engineering with a focus on mechanics. Our examination involves a manually crafted exam encompassing 126 multiple-choice questions, spanning various aspects of mechanics courses, including Fluid Mechanics, Mechanical Vibration, Engineering Statics and Dynamics, Mechanics of Materials, Theory of Elasticity, and Continuum Mechanics. Three LLMs, including ChatGPT (GPT-3.5), ChatGPT (GPT-4), and Claude (Claude-2.1), were subjected to evaluation against engineering faculties and students with or without mechanical engineering background. The findings reveal GPT-4's superior performance over the other two LLMs and human cohorts in answering questions across various mechanics topics, except for Continuum Mechanics. This signals the potential future improvements for GPT models in handling symbolic calculations and tensor analyses. The performances of LLMs were all significantly improved with explanations prompted prior to direct responses, underscoring the crucial role of prompt engineering. Interestingly, GPT-3.5 demonstrates improved performance with prompts covering a broader domain, while GPT-4 excels with prompts focusing on specific subjects. Finally, GPT-4 exhibits notable advancements in mitigating input bias, as evidenced by guessing preferences for humans. This study unveils the substantial potential of LLMs as highly knowledgeable assistants in both mechanical pedagogy and scientific research.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
