Successful software grows, and so does its test suite. Increasingly more of our customers have test suites that take several hours of even days to execute in full. However, the longer the time between the introduction of a bug during development, and its discovery through a test failure, the harder it gets to pinpoint the change that introduced this bug. The longer the test execution time, the larger the pain for all involved developers and testers.

If the execution of the entire test suite takes too long, a simple solution is to only execute a subset of the suite that require less time. If this subset is chosen well, its execution can find the majority of the bugs very quickly. The remaining tests can then still be executed every night / week to detect the remaining bugs. This way, we achieve rapid bug discovery times for most new bugs even for large, slow test suites.

However, how well this works in practice depends on how we select the subset of tests that gets executed more frequently. In recent years, we have experimented with many various selection techniques, both in research projects and together with our customers. Two work very well: Test-Impact-Analysis and Pareto-Optimization of test suites.

In this talk, I present both approaches. I share their basic ideas and our research findings. I also present how they can be applied to the Robot Framework and give examples from the Robot Framework and our customers.