基于航空航天制造专业知识的LLM评估：自动化生成与多模型问答
LLM Evaluation Based on Aerospace Manufacturing Expertise: Automated Generation and Multi-Model Question Answering
AI总结：
本文针对大型语言模型在航空航天制造领域易产生幻觉的问题，基于经典教材提取关键信息并自动生成多选题，通过多模型问答实验评估其准确性，发现现有模型专业能力亟待提高，为该领域应用提供了理论与实践指导。
摘要：
航空航天制造对技术参数的精度要求极高。GPT-4和QWen等大型语言模型（LLMs）在自然语言处理中的卓越表现，引发了业界将其应用于工艺设计、材料选择和刀具信息检索等任务的兴趣。然而，LLMs在专业领域容易产生“幻觉”，生成不准确或虚假的信息，对航空航天产品质量和飞行安全构成重大风险。本文提出了一套专为航空航天制造领域LLMs量身定制的评估指标，旨在通过分析模型在回答基于专业知识的问题时的表现来评估其准确性。首先，通过对经典航空航天制造教科书和指南进行深入文本分析，提取关键信息。随后，利用LLM生成技术，精心构建了具有不同难度且包含多个正确答案的多项选择题。接着，采用不同的LLM模型回答这些问题，并记录其准确率。实验结果表明，LLMs在航空航天专业知识方面的能力亟待提高。本研究为LLMs在航空航天制造中的应用提供了理论基础和实践指导，填补了该领域的关键空白。
英文摘要：
Aerospace manufacturing demands exceptionally high precision in technical parameters. The remarkable performance of Large Language Models (LLMs), such as GPT-4 and QWen, in Natural Language Processing has sparked industry interest in their application to tasks including process design, material selection, and tool information retrieval. However, LLMs are prone to generating "hallucinations" in specialized domains, producing inaccurate or false information that poses significant risks to the quality of aerospace products and flight safety. This paper introduces a set of evaluation metrics tailored for LLMs in aerospace manufacturing, aiming to assess their accuracy by analyzing their performance in answering questions grounded in professional knowledge. Firstly, key information is extracted through in-depth textual analysis of classic aerospace manufacturing textbooks and guidelines. Subsequently, utilizing LLM generation techniques, we meticulously construct multiple-choice questions with multiple correct answers of varying difficulty. Following this, different LLM models are employed to answer these questions, and their accuracy is recorded. Experimental results demonstrate that the capabilities of LLMs in aerospace professional knowledge are in urgent need of improvement. This study provides a theoretical foundation and practical guidance for the application of LLMs in aerospace manufacturing, addressing a critical gap in the field.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
