2025-02-13 –, RoboCon
The World of Data processing is evolving like never before. ETL is one of the wide used Data Integration process. Testing of these processes manually be time consuming. Hence Automation of such ETL Testing tasks plays a vital role. In this talk we would seeing how we have harnessed RobotFramework and Automated the Testing of Complex ETL Ecosystems.
The talk will include the whole ETL Test Automation Process. It elaborates the below processes which we followed:
1. Requirement Gathering
2. Test Data Preparation
3. Setting up application configuration
4. Triggering Application Jobs with the flexible environment to run the tests
5. Data comparison methodology
Testing technique ranges from testing of direct Transformed data sets to testing complex application Jobs involving Slowly Changing Dimensions etc.
With the above setup we have automated validation of 5000+ Data Attributes which runs as Regression for every Fortnight Releases.
Tech Stack: Spark, Scala and Hadoop
Test Automation Tool Used: Robot Framework with Python
Test Management Tool: qTest
Test Data Used: Masked Synthetic Data along with Relational Data Integrity
In Person
Categorize / Tags:Test Automation, ETL, BI Testing, Big Data Testing
Lessons Learned:How to develop a large scale Automation solution for the given Manual Testing Process
Describe your intended audience:Users who use Robot Framework for Test Automation of BI/ETL Jobs
Is this suitable for ..?:Beginner RF User, Intermediate RF User, Advanced RF User
An Aspiring Leader and a fullstack Test Automation Architect/Manager with 13+ years of experience in the IT Industry. Has wide range of Industrial experience including Banking, Retail, Airline and Market Research.
An expert in Architecting Test Automation Frameworks, ETL Test Automation, Big Data Automation and Robot Framework Testing
An expert in Automating Front-end, Web Services/API (SOAP and REST) and Backend application testing. Well versed with Python and Java programming languages. Highly experienced in CICD tools like Jenkins.
ISTQB Certified and successfully management a multi-vendor Team of Peak size 15 (located across 4 countries) towards multiple successful deliveries.