Recently AJQ has been working with a government organisation that had responsibility for creating state- wide infrastructure on a high-profile project. The tests involved mission critical patient and medical records.
They wanted to explore the possibility of using automation to improve the speed and scope of functional testing. They also wanted to extend coverage, improve quality and to reduce the duration of the release cycle, freeing up their functional test team to concentrate on higher priority issues.
First, we sat down with the project team to understand the risks, requirements and technology that had been used to create the solution.
The application’s design and the critical nature of the data meant that there had to be multiple validations along its journey. This was to ensure correct transformation and the correct information appearing to the end user in multiple locations in the portal.
This process, when performed by a functional tester, took so long that compromises were required in the coverage for which tests were performed as there were initially a large volume of tests required.
The department was certainly not willing to compromise on the quality or test coverage but needed a more efficient way of completing the test coverage and still meeting the deadline.
They realised by simply ramping up the existing team it would create additional challenges including time to educate the team on the system, additional office space and time to on-board new resources.
Other requirements included;
- The tool had to be robust and easy to use
- The results had to be easy to view and share
- The solution had to be available in all test environments
- The solution had to reduce the workload on the functional test team
- The tool had to be scalable as the number of tests will increase 10-fold
- The tests need to be run by the client’s team with limited technical knowledge of automation
We agreed to start with a Proof of Concept (PoC) to ensure that the tools used would be a good fit and the client could see the benefits and the quality of the work before we proceeded with full automation framework.
After a successful PoC, we created a framework that was flexible and extensible. Allowing integration of testing mechanisms at all levels of the application and adaptation to the customers changing needs.
The tests are run from a Jenkins web interface which required the test team to input only a few basic details to run the tests, such as environment and type of test.
It includes a rich reporting interface that produces reports that could be understood and shared by all members of the project team.
We also included a semi-automated defect logging system that captured all details of the failure (screenshots, error messages, test data) and allowed the tester to review prior to being automatically logged to the defect application.
As part of the project, training sessions were developed and ran, user handbooks were produced on how to use the automation tool. Thus, opening a line of communication to the teams utilising the tool to ensure they made the most of their investment.
Finally, by using Jenkins agents and Selenium Grid we ensured that if demand grew then the application would scale linearly with the addition of extra automation test servers.
The result of this project meant that what previously would have taken one member of the functional test team 60 days to achieve full coverage could be achieved in 24 hours. This was via the automation framework and utilisation of the machines available at that time. Over 200 unique tests were automated allowing for over 2000 variations to be carried out through data pools feeding into the automated scripts.
This meant an increase in the quality of the testing, much quicker test execution and faster project delivery.