The quality assessment of Artificial Intelligence (AI) systems is a fundamental challenge due to their inherently probabilistic nature. Standards such as ISO/IEC 25059 provide a quality model, but they lack practical and statistically robust methods for assessing functional correctness. This paper proposes and evaluates the Statistical Confidence in Functional Correctness (SCFC) approach, which seeks to fill this gap by connecting business requirements to a measure of statistical confidence that considers both the model's average performance and its variability. The approach consists of four steps: defining quantitative specification limits, performing stratified and probabilistic sampling, applying bootstrapping to estimate a confidence interval for the performance metric, and calculating a capability index as a final indicator. The approach was evaluated through a case study on two real-world AI systems in industry involving interviews with AI experts. Valuable insights were collected from the experts regarding the utility, ease of use, and intention to adopt the methodology in practical scenarios. We conclude that the proposed approach is a feasible and valuable way to operationalize the assessment of functional correctness, moving the evaluation from a point estimate to a statement of statistical confidence.
Beyond Functional Correctness: Design Issues in AI IDE-Generated Large-Scale Projects
School of Computer Science, Wuhan University(武汉大学计算机学院);School of Mathematical and Computational Sciences, Massey University(梅西大学数学与计算科学学院);School of Computer Science, Nanjing University of Science and Technology(南京理工大学计算机科学与工程学院);School of Computer Science, Central China Normal University(华中师范大学计算机学院);School of Computing Technologies, RMIT University(RMIT大学计算技术学院)
RealSec-bench: A Benchmark for Evaluating Secure Code Generation in Real-World Repositories
Sun Yat-sen University(中山大学);Nanyang Technological University(南洋理工大学);Alibaba Group(阿里巴巴集团);The State Key Laboratory of Blockchain and Data Security, Zhejiang University(浙江大学区块链与数据安全国家重点实验室)
Beyond Functional Correctness: Exploring Hallucinations in LLM-Generated Code
2 State Key Laboratory of Complex \& Critical Software Environment (SKLCCSE) School of Computer Science;Engineering, Beihang University, Beijing, China;3 School of Software, Beihang University, Beijing, China;4 School of Computer Science;Technology, Shandong University, Qingdao, China;5 Huawei Cloud Computing Technologies Co., Ltd, China