arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2211.16587cs.SE
收藏

基于语言基数的模型推理准确性严格评估

Rigorous Assessment of Model Inference Accuracy using Language Cardinality

Donato Clun, Donghwan Shin, Antonio Filieri, Domenico Bianculli

AI总结:

针对现有模型推理准确性评估依赖统计估计、结果有偏差且不稳定的问题,本文提出基于解析组合学的确定性评估方法,并用基准实验验证了其一致性与适用性。

摘要:

有限状态自动机等模型通过捕获软件系统执行期间可观测的事件序列,被广泛用于对软件系统的行为进行抽象。然而,模型在实践中很少存在,即便存在也很容易过时;此外,手动构建和维护模型成本高昂且容易出错。因此,人们提出了多种从执行轨迹自动构建模型的模型推理方法来解决这些问题。\n 然而,对推理得到的模型开展系统、可靠的准确性评估仍然是一个未解决的问题。即便给定参考模型,大多数现有的模型准确性评估方法也可能返回具有误导性且有偏差的结果。这主要是因为它们依赖于对有限数量随机生成轨迹的统计估计,会给估计带来可避免的不确定性,且对随机轨迹生成过程的参数较为敏感。\n 本文通过开发一种基于解析组合学的系统化方法来解决这一问题,该方法用确定性准确性度量替代统计估计,从而最大限度降低模型准确性评估中的偏差与不确定性。我们通过用成熟的规约挖掘基准中的参考模型,评估前沿推理工具推理得到的模型的准确性,通过实验证明了我们方法的一致性与适用性。

英文摘要:

Models such as finite state automata are widely used to abstract the behavior of software systems by capturing the sequences of events observable during their execution. Nevertheless, models rarely exist in practice and, when they do, get easily outdated; moreover, manually building and maintaining models is costly and error-prone. As a result, a variety of model inference methods that automatically construct models from execution traces have been proposed to address these issues. However, performing a systematic and reliable accuracy assessment of inferred models remains an open problem. Even when a reference model is given, most existing model accuracy assessment methods may return misleading and biased results. This is mainly due to their reliance on statistical estimators over a finite number of randomly generated traces, introducing avoidable uncertainty about the estimation and being sensitive to the parameters of the random trace generative process. This paper addresses this problem by developing a systematic approach based on analytic combinatorics that minimizes bias and uncertainty in model accuracy assessment by replacing statistical estimation with deterministic accuracy measures. We experimentally demonstrate the consistency and applicability of our approach by assessing the accuracy of models inferred by state-of-the-art inference tools against reference models from established specification mining benchmarks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑