探索大型语言模型在Simulink-Stateflow变异体生成中的潜力
Exploring the Potential of Large Language Models in Simulink-Stateflow Mutant Generation
AI总结：
本文利用大型语言模型生成高质量的Simulink-Stateflow变异体，显著提高了生成效率和变异体质量。
摘要：
变异分析是一种评估测试套件充分性的强大技术，但传统方法在生成冗余、等价或不可执行的变异体方面存在困难。这些挑战在Simulink-Stateflow模型中尤为突出，因为这些模型具有分层结构，集成了连续动态和离散事件行为，并广泛应用于安全关键的物理系统（CPSs）。尽管已有研究探索了机器学习和手动工程化的变异运算符，但这些方法仍然受限于有限的训练数据和可扩展性问题。受最近大型语言模型（LLMs）进展的启发，我们研究其生成高质量、领域特定的Simulink-Stateflow模型变异体的潜力。我们开发了一个自动化流程，将Simulink-Stateflow模型转换为结构化的JSON表示，并系统评估了不同变异和提示策略在八种最先进的LLMs中的表现。通过涉及38,400个LLM生成变异体的全面实证研究，涵盖四个Simulink-Stateflow模型，我们证明LLMs生成变异体的速度比手动工程化的变异基线快13倍，同时生成的等价和重复变异体显著减少，并且始终达到更高质量的变异体。此外，我们的分析揭示了少样本提示结合低至中等温度值能产生最佳结果。我们提供了一个开源原型工具，并释放了完整的数据集以促进可重复性和推动该领域的未来研究。
英文摘要：
Mutation analysis is a powerful technique for assessing test-suite adequacy, yet conventional approaches suffer from generating redundant, equivalent, or non-executable mutants. These challenges are particularly amplified in Simulink-Stateflow models due to the hierarchical structure these models have, which integrate continuous dynamics with discrete-event behaviors and are widely deployed in safety-critical Cyber-Physical Systems (CPSs). While prior work has explored machine learning and manually engineered mutation operators, these approaches remain constrained by limited training data and scalability issues. Motivated by recent advances in Large Language Models (LLMs), we investigate their potential to generate high-quality, domain-specific mutants for Simulink-Stateflow models. We develop an automated pipeline that converts Simulink-Stateflow models to structured JSON representations and systematically evaluates different mutation and prompting strategies across eight state-of-the-art LLMs. Through a comprehensive empirical study involving 38,400 LLM-generated mutants across four Simulink-Stateflow models, we demonstrate that LLMs generate mutants up to 13x faster than a manually engineered mutation-based baseline while producing significantly fewer equivalent and duplicate mutants and consistently achieving superior mutant quality. Moreover, our analysis reveals that few-shot prompting combined with low-to-medium temperature values yields optimal results. We provide an open-source prototype tool and release our complete dataset to facilitate reproducibility and advance future research in this domain.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
