Testing is widely recognized as an important stage of the software development lifecycle. Effective software testing can provide benefits such as bug finding, preventing regressions, and documentation. In terms of documentation, unit tests express a unit's intended functionality, as conceived by the developer. A test oracle, typically expressed as an condition, documents the intended behavior of a unit under a given test prefix. Synthesizing a functional test oracle is a challenging problem, as it must capture the intended functionality rather than the implemented functionality.
In this paper, we propose TOGA (a neural method for Test Oracle GenerAtion), a unified transformer-based neural approach to infer both exceptional and assertion test oracles based on the context of the focal method. Our approach can handle units with ambiguous or missing documentation, and even units with a missing implementation. We evaluate our approach on both oracle inference accuracy and functional bug-finding. Our technique improves accuracy by 33\% over existing oracle inference approaches, achieving 96\% overall accuracy on a held out test dataset. Furthermore, we show that when integrated with a automated test generation tool (EvoSuite), our approach finds 57 real world bugs in large-scale Java programs, including 30 bugs that are not found by any other automated testing method in our evaluation.
MASTOR: A Multi-Agent Approach to Semantic Test Oracle Generation for RESTful APIs
Macau University of Science and Technology（澳门科技大学）；Macau University of Science and Technology Zhuhai MUST Science and Technology Research Institute（澳门科技大学珠海MUST科学技术研究院）；Beihang University（北京航空航天大学）；China University of Mining and Technology（中国矿业大学）
Escaping the Self-Repair Trap: Improving Test Oracle Generation via Dual-Context Awareness
Beihang University（北京航空航天大学）；National College for Excellent Engineers（卓越工程师学院）；Qingdao Research Institute（青岛研究院）；Hangzhou Innovation Institute（杭州创新研究院）
Machine learning may enable the automated generation of test oracles. We have characterized emerging research in this area through a systematic literature review examining oracle types, researcher goals, the ML techniques applied, how the generation process was assessed, and the open research challenges in this emerging field. Based on a sample of 22 relevant studies, we observed that ML algorithms generated test verdict, metamorphic relation, and - most commonly - expected output oracles. Almost all studies employ a supervised or semi-supervised approach, trained on labeled system executions …
Correct Tests Are Not Enough: Measuring and Training Oracle Conversion in Specification-Based Test Generation
Chengdu Institute of Computer Applications, Chinese Academy of Sciences（中国科学院成都计算机应用研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Institute of Multidisciplinary Research for Advanced Materials (IMRAM), Tohoku University（东北大学先进材料多学科研究所）；Artificial Intelligence Research Institute, Shenzhen University of Advanced Technology（深圳先进技术研究院人工智能研究所）
Game-like programs have become increasingly popular in many software engineering domains such as mobile apps, web applications, or programming education. However, creating tests for programs that have the purpose of challenging human players is a daunting task for automatic test generators. Even if test generation succeeds in finding a relevant sequence of events to exercise a program, the randomized nature of games means that it may neither be possible to reproduce the exact program behavior underlying this sequence, nor to create test assertions checking if observed randomized game behavior …
Automated test generation has helped to reduce the cost of software testing. However, developing effective test oracles for these automatically generated test inputs is a challenging task. Therefore, most automated test generation tools use trivial oracles that reduce the fault detection effectiveness of these automatically generated test cases. In this work, we provide results of an empirical study showing that utilizing metamorphic relations can increase the fault detection effectiveness of automatically generated test cases.
MANGO: Automated Multi-Agent Test Oracle Generation for Vision-Language-Action Models
Mondragon University（蒙得龙大学）；Simula Research Laboratory（Simula研究实验室）；University of Ottawa（渥太华大学）；Research Ireland Lero Centre for Software（爱尔兰Lero软件研究中心）
LLM-Based Test Generation: Information Sources, Generation Strategies, and Quality Evidence
Chengdu Institute of Computer Applications, Chinese Academy of Sciences（中国科学院成都计算机应用研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Institute of Multidisciplinary Research for Advanced Materials (IMRAM), Tohoku University（东北大学先进材料多学科研究所）；Artificial Intelligence Research Institute, Shenzhen University of Advanced Technology（深圳先进技术研究院人工智能研究所）
Auditing and Decomposing Feedback-Driven Evolution in LLM Test Generation under the Oracle Problem
Chengdu Institute of Computer Applications, Chinese Academy of Sciences（中国科学院成都计算机应用研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Institute of Multidisciplinary Research for Advanced Materials (IMRAM), Tohoku University（东北大学先进材料多学科研究所）；Artificial Intelligence Research Institute, Shenzhen University of Advanced Technology（深圳先进技术研究院人工智能研究所）
Automated model-based test generation presents a viable alternative to the costly manual test creation currently employed for regression testing of web apps. However, existing model inference techniques rely on threshold-based whole-page comparison to establish state equivalence, which cannot reliably identify near-duplicate web pages in modern web apps. Consequently, existing techniques produce inadequate models for dynamic web apps, and fragile test oracles, rendering the generated regression test suites ineffective. We propose a model-based test generation technique, FRAGGEN, that eliminate…
Merging other branches into the current working branch is common in collaborative software development. However, developers still heavily rely on the textual merge tools to handle the complicated merge tasks. The latent semantic merge conflicts may fail to be detected and degrade the software quality. Regression testing is able to prevent regression faults and has been widely used in real-world software development. However, the merged software may fail to be well examined by rerunning the existing whole test suite. Intuitively, if the test suite fails to cover the changes of different branche…
Test oracle generation in non-regression testing is a longstanding challenge in software engineering, where the goal is to produce oracles that can accurately determine whether a function under test (FUT) behaves as intended for a given input. In this paper, we introduce Nexus, a novel multi-agent framework to address this challenge. Nexus generates test oracles by leveraging a diverse set of specialized agents that synthesize test oracles through a structured process of deliberation, validation, and iterative self-refinement. During the deliberation phase, a panel of four specialist agents, e…
Model transformations are the cornerstone of Model-Driven Engineering, and provide the essential mechanisms for manipulating and transforming models. Checking whether the output of a model transformation is correct is a manual and error-prone task, this is referred to as the oracle problem in the software testing literature. The correctness of the model transformation program is crucial for the proper generation of its output, so it should be tested. Metamorphic testing is a testing technique to alleviate the oracle problem consisting on exploiting the relations between different inputs and ou…
Metamorphic testing has become one mainstream technique to address the notorious oracle problem in software testing, thanks to its great successes in revealing real-life bugs in a wide variety of software systems. Metamorphic relations, the core component of metamorphic testing, have continuously attracted research interests from both academia and industry. In the last decade, a rapidly increasing number of studies have been conducted to systematically generate metamorphic relations from various sources and for different application domains. In this article, based on the systematic review on t…
Automated Trustworthiness Oracle Generation for Machine Learning Text Classifiers
Monash University（莫纳什大学）；University of Auckland（奥克兰大学）；Royal Melbourne Institute of Technology（皇家墨尔本理工学院）；Joint Research Centre at the European Commission（欧洲委员会联合研究中心）
There have been many studies on automated test generation for mobile Graphical User Interface (GUI) applications. These studies successfully demonstrate how to detect fatal exceptions and achieve high code and activity coverage with fully automated test generation engines. However, it is unclear how many GUI functions these engines manage to test. Furthermore, these engines implement only implicit test oracles. We propose Fully Automated Reinforcement LEArning-Driven Specification-Based Test Generator for Android (FARLEAD-Android). FARLEAD-Android accepts a GUI-level formal specification as a …
Metamorphic testing is a well known approach to tackle the oracle problem in software testing. This technique requires the use of source test cases that serve as seeds for the generation of follow-up test cases. Systematic design of test cases is crucial for the test quality. Thus, source test case generation strategy can make a big impact on the fault detection effectiveness of metamorphic testing. Most of the previous studies on metamorphic testing have used either random test data or existing test cases as source test cases. There has been limited research done on systematic source test cas…
BackendForge: Benchmarking Agentic End-to-End Code Generation with Backend Services
Key Lab of HCST (PKU), MOE（北京大学计算机科学与技术国家重点实验室；软件学院，北京大学）；SCS, Peking University（北京通明湖信息技术应用创新中心）；Beijing Tongming Lake Information Technology Application Innovation Center (TLAIC)（复旦大学高级计算系统研究所）；Fudan University Institute of Systems for Advanced Computing（上海开放计算系统研究所）；Shanghai Institute of Systems for Open Computing
University of Stuttgart（斯图加特大学）；Federal University of Pernambuco（伯南布哥联邦大学）
Few-shot learning with large-scale, pre-trained language models is a powerful way to answer questions about code, e.g., how to complete a given code example, or even generate code snippets from scratch. The success of these models raises the question whether they could serve as a basis for building a wide range code generation tools. Traditionally, such tools are built manually and separately for each task. Instead, few-shot learning may allow to obtain different tools from a single pre-trained language model by simply providing a few examples or a natural language description of the expected …
Metamorphic testing (MT) is a general approach for the testing of a specific kind of software systems -- so-called ``non-testable'', where the ``classical'' testing approaches are difficult to apply. MT is an effective approach for addressing the test oracle problem and test case generation problem. The test oracle problem is when it is difficult to determine the correct expected output of a particular test case or to determine whether the actual outputs agree with the expected outcomes. The core concept in MT is metamorphic relations (MRs) which provide formal specification of the system unde…
