2022-05-19 –, Main Hall
Tests provide feedback, but the pass/fail/skip status of the current test run is not the only way test can provide value for the users. This talk will show multiple ways of looking at the data and what other value testing can provide.
Testing provides valuable feedback for teams and people. Most of the time people look at the current status of the test automation execution, either from a radiator or from their CI systems. But tests can provide also other valuable feedback than just looking at the current pass/fail count. This talk shows three different ways how people can benefit when looking their test execution result with a different views: using coverage information, flaky test detection and long term historical data.
Coverage information is usually displayed as a number and teams usually aim to keep the covereage number as high as possible. But coverage is more than number in the teams dashboard. With Browser library development, coverage reporting has helped me to focus the testing and find places where more testing is needed. Coverage report, with combination of other data, helps me to improve the testing effort.
Flaky test can ruin trust for any test automation process and when flaky tests happen often enough people do not anymore value the test automation effort. By showing which tests fail randomly most often, information can help focusing the efforts for fixing the root cause of the flaky tests. This part of the talk shows how flaky test detection is integrated in Browser library CI, what information it provides and what other data could be used in flaky test identification.
When test results, with metadata, are saved for a long term in to a visualization service, user can ask questions from the test data. Example how many tests team is running, how pass/fail/skip ratio is looking, what version is good to release and many more. Also test result analysis can affect prioritization of the fixes, finding important/unique failures and many more. This talk shows example how test data visualization has helped me at F-Secure.
Tests can talk in many ways: in short, medium land long term. Saving and processing the data from tests can help us to understand new and exiting things.
Tatu Aalto works at F-Secure as Senior Software Developer in Test with Windows anti-virus client team. He has +10 years experience with Robot Framework and more than 20 years experience in testing at general. Tatu maintains several libraries or tools in the Robot Framework ecosystem, example PyhtonLibCore and is one of the core team member in the Browser library.