arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2607.06004cs.SE
收藏

大语言模型对软件工程术语的理解不可靠

Large Language Models Have Unreliable Understanding of Software Engineering Terminology

Faculty of Computer Science;Mathematics University of Passau Passau, Germany

Huzaifa Ejaz, Fabian C. Peña, Steffen Herbold

AI总结:

研究大语言模型对软件工程术语的理解程度,通过用正确及伪造定义提示,测量分类准确率与推理令牌合理性,发现多数模型虽能检测伪造定义,但存在拒绝正确定义的偏差,明确推理未持续改善结果,揭示了LLMs在术语理解上的问题。

摘要:

大语言模型(LLMs)在软件工程(SE)中应用日益广泛,但尚无系统研究确定其对标准化SE术语的理解程度。缺乏理解会导致沟通误解。本文研究了先进LLMs识别ISO/IEC/IEEE 24765:2017系统与软件工程 - 词汇表中定义是否正确的能力。用正确和伪造定义提示LLMs,伪造包括语义和结构方面。测量分类准确率及推理令牌合理性。多数LLMs能高精度检测伪造定义,但也拒绝很多正确定义,存在系统性拒绝偏差。明确推理未持续改善结果,甚至可能因过度思考阻碍性能。研究表明LLMs在许多SE任务中表现令人印象深刻,但在理解其对SE影响包括术语一致使用方面仍有基本问题。

英文摘要:

Large Language Models (LLMs) are increasingly used in software engineering (SE), yet there is no systematic study that determines to which degree these LLMs actually understand standardized SE terminology. Lack of such understanding can lead to miscommunication and misunderstanding, both by LLMs consuming text but also by human-developers acting on LLM-generated text. Within this paper, we investigate to which degree state-of-the-art LLMs are able to identify whether definitions from the ISO/IEC/IEEE 24765:2017 Systems and Software Engineering - Vocabulary are correct. We prompt LLMs both with correct definitions, as well as systematically falsified definitions. The falsifications are both semantic (substitution of key terms) and structural (removing critical information). We measure both classification accuracy and whether reasoning tokens generated by the LLMs make sense with respect to understanding the definition. While most LLMs detect falsified definitions with high accuracy, they also reject many correct definitions, indicating a systematic rejection bias rather than genuine discriminative understanding. Explicit reasoning does not consistently improve results and may even hinder performance through over-thinking. Our work demonstrates that while the performance of LLMs (including their agentic use) in many SE tasks is impressive, there are still fundamental issues to understand how this will impact SE, including the consistent use of terminology.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑