ChatGPT 支持任务工程中问题表述任务能力的实证探索及其性能变异性的记录
An Empirical Exploration of ChatGPT's Ability to Support Problem Formulation Tasks for Mission Engineering and a Documentation of its Performance Variability
AI总结：
本文通过 NASA 太空任务设计挑战实证评估 ChatGPT-3.5 在任务工程问题表述中识别利益相关者的能力，发现其对人本利益相关者识别较好但对环境因素识别差、抽象层次保持不足且并行输出变异性大，表明其可靠性有限。
摘要：
系统工程（SE）正随着生成式人工智能（AI）的可用性以及对系统之系统视角的需求而不断发展，这一视角在美国国防部的任务工程（ME）框架下得到正式化。表述 ME 问题具有挑战性，因为它们是开放式练习，涉及将定义不清的问题转化为适合工程开发的定义明确的问题。AI 能在多大程度上辅助问题表述目标仍有待观察。为此，本文探讨了多用途大语言模型（LLM）在支持 ME 问题表述任务中的质量和一致性，特别聚焦于利益相关者识别。我们确定了一个相关的参考问题，即 NASA 太空任务设计挑战，并记录了 ChatGPT-3.5 执行利益相关者识别任务的能力。我们执行了多次并行尝试，并对 LLM 输出进行了定性评估，重点关注其质量和变异性。我们的发现呈现出一幅微妙的图景。我们发现，该 LLM 在识别人本利益相关者方面表现良好，但在识别外部系统和环境因素方面表现不佳，尽管我们明确努力将这些因素纳入考虑。此外，LLM 在保持所需抽象层次方面存在困难，并倾向于产生针对具体解决方案的输出，这些输出不适合用于问题表述。更重要的是，我们记录了并行线程之间的巨大变异性，强调 LLM 输出应谨慎使用，理想情况下应采用对其能力的随机视角。总体而言，我们的发现表明，尽管 ChatGPT 可以减少部分专家工作量，但其缺乏一致性和领域理解可能限制其在问题表述任务中的可靠性。
英文摘要：
Systems engineering (SE) is evolving with the availability of generative artificial intelligence (AI) and the demand for a systems-of-systems perspective, formalized under the purview of mission engineering (ME) in the US Department of Defense. Formulating ME problems is challenging because they are open-ended exercises that involve translation of ill-defined problems into well-defined ones that are amenable for engineering development. It remains to be seen to which extent AI could assist problem formulation objectives. To that end, this paper explores the quality and consistency of multi-purpose Large Language Models (LLM) in supporting ME problem formulation tasks, specifically focusing on stakeholder identification. We identify a relevant reference problem, a NASA space mission design challenge, and document ChatGPT-3.5's ability to perform stakeholder identification tasks. We execute multiple parallel attempts and qualitatively evaluate LLM outputs, focusing on both their quality and variability. Our findings portray a nuanced picture. We find that the LLM performs well in identifying human-focused stakeholders but poorly in recognizing external systems and environmental factors, despite explicit efforts to account for these. Additionally, LLMs struggle with preserving the desired level of abstraction and exhibit a tendency to produce solution specific outputs that are inappropriate for problem formulation. More importantly, we document great variability among parallel threads, highlighting that LLM outputs should be used with caution, ideally by adopting a stochastic view of their abilities. Overall, our findings suggest that, while ChatGPT could reduce some expert workload, its lack of consistency and domain understanding may limit its reliability for problem formulation tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
