Coincidental correctness (CC) arises when a defective program produces the correct output despite the fact that the defect within was exercised. Researchers have recognized the negative impact of coincidental correctness, and the authors have previously conducted a study demonstrating its prevalence in test suites. However, that study was limited to system tests and small subjects seeded with artificial defects. In this paper, we conduct a wider scope study of CC that addresses the following research questions in the context of the Defects4J benchmark:
RQ1: Is CC prevalent in Defects4J?
RQ2: Is CC affected by the testing levels in Defects4J?
RQ3: Do CC tests induce peculiar infection paths in Defects4J?
RQ4: Are the infections likely to be nullified within or outside the buggy method? ....
Beyond Correctness: Benchmarking Multi-dimensional Code Generation for Large Language Models
University of Chinese Academy of Sciences（中国科学院大学）；Institute of Software, Chinese Academy of Sciences（中国科学院软件研究所）；Chinese Information Processing Laboratory（中文信息处理实验室）；State Key Laboratory of Computer Science（计算机科学国家重点实验室）；Key Laboratory of System Software, Chinese Academy of Sciences（中国科学院系统软件重点实验室）
RealSec-bench: A Benchmark for Evaluating Secure Code Generation in Real-World Repositories
Sun Yat-sen University（中山大学）；Nanyang Technological University（南洋理工大学）；Alibaba Group（阿里巴巴集团）；The State Key Laboratory of Blockchain and Data Security, Zhejiang University（浙江大学区块链与数据安全国家重点实验室）
DSEffi-Bench: Demystifying Large Language Models' Capability in Efficient Data Science Code Generation
Key Lab of HCST (PKU), MOE（北京大学计算机辅助设计与图形学教育部重点实验室；北京大学信息科学技术学院）；SCS, Peking University, China（北京大学深圳研究生院电子与计算机工程学院）；School of Electronic and Computer Engineering, PKU Shenzhen Graduate School, China（新加坡国立大学）；National University of Singapore, Singapore（中国科学院软件研究所）；Institute of Software, Chinese Academy of Sciences, Beijing, China（伦敦国王学院）；King’s College London, London, United Kingdom
