A new project seeks to make effective testing more compatible with expeditious releases
In August 2021, a team of graduate students from IDC Herzliya, a leading research college in Israel, began working on a test case prioritization (TCP) project under the guidance of senior engineers from Red Hat.
The goal of the TCP project is to create a novel tool based on machine learning (ML) that solves the test case prioritization problem in software regression testing. Automatic regression testing is a crucial step of any CI/CD pipeline. The primary purpose of this testing is to detect bugs and defects introduced by recent changes as early as possible while keeping verification costs low. The ability to perform regression testing efficiently and effectively (i.e., within a small time frame, yet catching the majority of bugs) would allow developers to rapidly deliver reliable software updates to users.
Unfortunately, the regression testing process in modern large-scale software products tends to be too complex and cumbersome to meet this objective. As the size of software increases, the test suite also grows bigger and requires more time and resources to be fully executed. In many cases, the time to run the entire test suite can reach three or four days or even a week. Consequently, executing all available tests during the CI/CD regression testing procedure is highly impractical and, in many cases, completely infeasible.
To address this issue, TCP methods have emerged. TCP aims to order a given set of test cases such that the earlier a test appears in the resulting order, the higher the probability that this test will detect a bug or a fault introduced by the given code change. Provided such an ordering exists for the entire test suite, the regression testing procedure can iterate over it starting from the beginning and advancing until a predefined limit on the maximal testing time (say, one minute) is reached. Because the most significant test cases are executed first, the chances of early fault detection rise even when executing only a negligible part (say, 1%) of the entire test suite.
In recent years, TCP solutions have been adopted by major industry players and have spawned a wave of academic research projects. However, the full potential of TCP in regression testing is yet to be explored. As of now, the most widely used approaches are mainly based on heuristic search strategies and/or code coverage methods. In contrast, methods based on machine learning in general and deep learning in particular are barely explored. Closing this gap and devising a ML-based TCP tool is the primary goal of this project.
The project team is now progressing towards the first major milestone: implementing an open source proof-of-concept prototype that replicates the state-of-the-art academic results in this area. This milestone is expected to be reached by November 2021. As an immediate next step, the team will be looking to improve the state of the art. To that end, a variety of machine learning and data mining methods will be considered.
Those interested in finding out more about the TCP project and/or looking for collaboration opportunities are kindly invited to contact Dr. Ilya Kolchinsky (ikolchin@redhat.com) or Gil Klein (gklein@redhat.com).