Instance Space Analysis of Search-Based Software Testing

Marcel Boehme A. Aleti Burak Turhan + 1 lainnya

1 Januari 2023

Automated test generators, such as search-based software testing (SBST) techniques are primarily guided by coverage information. As a result, they are very effective at achieving high code coverage. However, is high code coverage alone sufficient to detect bugs effectively? In this paper, we propose a new SBST technique, predictive many objective sorting algorithm (PreMOSA), which augments coverage information with defect prediction information to decide where to increase the test coverage in the class under test (CUT). Through an experimental evaluation using 420 labelled bugs on the Defects4J benchmark and using theoretical defect predictors, we demonstrate the improved effectiveness and efficiency of PreMOSA in detecting bugs when using any acceptable defect predictor, i.e., a defect predictor with recall and precision <inline-formula><tex-math notation="LaTeX">$\geq$</tex-math><alternatives><mml:math><mml:mo>≥</mml:mo></mml:math><inline-graphic xlink:href="perera-ieq1-3147008.gif"/></alternatives></inline-formula> 75%, compared to the state-of-the-art dynamic many objective sorting algorithm (DynaMOSA). PreMOSA detects up to 8.3% more labelled bugs on average than DynaMOSA when given a time budget of 2 minutes for test generation per CUT.

Alexandru Marginean N. Alshahwan M. Harman

1 April 2023

There have been rapid recent developments in automated software test design, repair and program improvement. Advances in artificial intelligence also have great potential impact to tackle software testing research problems. In this paper we highlight open research problems and challenges from an industrial perspective. This perspective draws on our experience at Meta Platforms, which has been actively involved in software testing research and development for approximately a decade. As we set out here, there are many exciting opportunities for software testing research to achieve the widest and deepest impact on software practice. With this overview of the research landscape from an industrial perspective, we aim to stimulate further interest in the deployment of software testing research. We hope to be able to collaborate with the scientific community on some of these research challenges.

A. Perera Burak Turhan A. Aleti + 1 lainnya

1 September 2020

Today, most automated test generators, such as search-based software testing (SBST) techniques focus on achieving high code coverage. However, high code coverage is not sufficient to maximise the number of bugs found, especially when given a limited testing budget. In this paper, we propose an automated test generation technique that is also guided by the estimated degree of defectiveness of the source code. Parts of the code that are likely to be more defective receive more testing budget than the less defective parts. To measure the degree of defectiveness, we leverage Schwa, a notable defect prediction technique.We implement our approach into EvoSuite, a state of the art SBST tool for Java. Our experiments on the Defects4J benchmark demonstrate the improved efficiency of defect prediction guided test generation and confirm our hypothesis that spending more time budget on likely defective parts increases the number of bugs found in the same time budget.

Tony Fan C. Menghi Federico Formica

22 Juli 2022

Search-based software testing (SBST) typically relies on fitness functions to guide the search exploration toward software failures. There are two main techniques to define fitness functions: (a) automated fitness function computation from the specification of the system requirements, and (b) manual fitness function design. Both techniques have advantages. The former uses information from the system requirements to guide the search toward portions of the input domain more likely to contain failures. The latter uses the engineers’ domain knowledge. We propose ATheNA, a novel SBST framework that combines fitness functions automatically generated from requirements specifications and those manually defined by engineers. We design and implement ATheNA-S, an instance of ATheNA that targets Simulink® models. We evaluate ATheNA-S by considering a large set of models from different domains. Our results show that ATheNA-S generates more failure-revealing test cases than existing baseline tools and that the difference between the runtime performance of ATheNA-S and the baseline tools is not statistically significant. We also assess whether ATheNA-S could generate failure-revealing test cases when applied to two representative case studies: one from the automotive domain and one from the medical domain. Our results show that ATheNA-S successfully revealed a requirement violation in our case studies.

Rui Lima A. M. R. da Cruz Jorge Ribeiro

1 Juni 2020

In the last few years Artificial Intelligence (AI) algorithms and Machine Learning (ML) approaches have been successfully applied in real-world scenarios like commerce, industry and digital services, but they are not a widespread reality in Software Testing. Due to the complexity of software testing, most of the work of AI/ML applied to it is still academic. This paper briefly presents the state of the art in the field of software testing, applying ML approaches and AI algorithms. The progress analysis of the AI and ML methods used for this purpose during the last three years is based on the Scopus Elsevier, web of Science and Google Scholar databases. Algorithms used in software testing have been grouped by test types. The paper also tries to create relations between the main AI approaches and which type of tests they are applied to, in particular white-box, grey-box and black-box software testing types. We conclude that black-box testing is, by far, the preferred method of software testing, when AI is applied, and all three methods of ML (supervised, unsupervised and reinforcement) are commonly used in black-box testing being the “clustering” technique, Artificial Neural Networks and Genetic Algorithms applied to “fuzzing” and regression testing.