So here is the day your team finally decides to start test automation. Having analyzed what can be automated, what is worth automating, and what systems you want to integrate with, you are now at the most critical question – how to do it. The market for test automation tools has grown tremendously in recent years. There are standard enterprise / off-the-shelf automation tools available. And a new range of tools based on artificial intelligence has also emerged. Nevertheless, approaches based on building your own framework are still relevant. The world of automation tools is a real game-changer and it’s rapidly developing both in terms of the range of products and their functionality. Today, we are covering the state of the art as of April 2022.
| Feature | Own Testing Framework | Corporate Tools | AI tools |
| Tailored to your needs | Perfectly fitted | Incomplete | Incomplete |
| Resistance to application changes | Low | Low | Very high |
| Application in other projects | Very small (no change in code) | Large | Large |
| Integration with other tools via API | Full | Partial | Partial |
| Option to develop new functionalities | Very large | Limited to manufacturer | Limited to manufacturer |
| Required technical knowledge to create tests | Necessary | Partial | Limited to minimum |
| Test development time | Long | Average | Short |
| License cost | Free | High | Free/Average |
The standard tools are well-known in most major companies and have been around for many years. They are often based on the Record & Playback automated test creation methodology, as well as on keywords. Thus, in order to start using these tools, it is normally only necessary to configure them at first, and then to program actions that are not inherently provided by the tool itself. As a result, test creation time can be considered average and in many cases longer than for AI tools.
It may also be required to have at least some programming skills. Undoubtedly, one of the biggest advantages of these tools is their genericity, i.e. the ability to be applied to many types of applications. In the vast majority, they also enable integration with external systems. However, it is worth bearing in mind that the integration is possible only with these systems, which were already designed by the tool’s manufacturer.
So, this brings us to the two biggest drawbacks of these tools. The aforementioned genericity causes these tools to rarely perfectly adjust to the requirements, and adding new functionalities to them is subject to the manufacturer’s decision only. Despite a large number of functionalities offered, it may still be difficult to change the graphical interface and thus will be forced to modify the automated tests.
Pricing is a common factor determining the purchase or resignation from enterprise tools. Licenses are usually made available as time-based subscriptions and their prices tend to be relatively high, which in the case of longer-term projects turns them into a substantial budget item for the testing team.
Frameworks with AI represent a new range of tools, conquering the test automation market for the past three years. They are intended to be an answer to the disadvantages of the enterprise tools outlined above. Indeed, their main benefit is the ability to self-adapt to the changing graphical interface of applications. AI tools also retain a certain degree of genericity, thanks to which it is feasible to use them in a wide array of projects.
The lack of code development is a further strong advantage, making them very user-friendly for those with limited programming skills. An additional extra edge comes from the very short time it takes to create automated tests. However, unfortunately, to some extent, they also suffer from the disadvantages of enterprise tools. Despite their flexibility, they still rarely meet the requirements set, and in the case of integration or adding new functionalities, one is still reliant on the vendor. On top of that, with a longer project duration and an increasing number of created tests, the whole automated testing framework becomes harder and harder to maintain, and despite an attractive price for a license, more and more effort is required to maintain tests over time, and that may lead to an inflated cost of maintaining the whole tool.
By far the strongest advantage of building a custom tool is its ideal adaptation to meet all needs and requirements. In this case, there are no concessions to be made, regardless of what type of application is to be tested, at what level, and how the test report is supposed to be laid out.
The scope of integration with other tools or adding new functionalities is entirely up to us and can be virtually unlimited. A side effect, on the other hand, is that at this point the testing framework gets so highly customized to the tested application that it becomes hard to use it in other projects unless changes are made. Plus, with no artificial intelligence mechanisms applied, as in the corporate tools, we may find our own tool to not be very robust to face any changes in the user interface.
Building your own tool means having to possess programming skills, which, depending on your team, can be a downside or an upside. A serious drawback of building your own test automation framework is also the initial longer time to create tests. This often continues until the in-house tool is mature, and then, in the long run, the time to create new automated tests is comparable to that of commercial tools. An undeniable asset comes with the lack of license fees for the tool, which makes it possible to balance the demand for the required additional upfront effort in its development. It is also worth emphasizing that a well-designed test framework is the most cost-effective option to maintain in terms of the growing number of automated tests.
We are occasionally approached by clients with a specific vision of how they want to automate testing. In those instances, we happen to point out the risks of the selected method. And very often we are contacted by those customers who realize that they need to automate tests due to time and budget but have no clue how to do so. Based on the competencies of our architects and test analysts we can assess the current test process, identify areas for automation, and eventually recommend optimal tools. We are not focused on one, favored technology, but instead strive to recommend the optimal version.
A good example is the use of Katalon Studio in a project for one of the largest Polish banks, which was chosen for its ability to automate tests in the BDD (Behavior Driven Development) methodology. Due to this, the further creation of new automated scenarios was made possible even for those with no programming expertise. Simultaneous support of both web and mobile application automated testing was also a sought-after asset of the tool.
A totally different approach was taken in the project for Voestalpine, where we created our own test framework. This was dictated by the need to ensure communication with specialized railway equipment, as well as to enable testing as early as at the level of particular system modules.
So, as you can clearly see, there is no definite answer to the question of which framework to choose. It all starts with the requirements, the application to be tested, the skills of the team, and the intended budget. At Solwit we are experienced in working with each of the tool types covered above.
We run thousands of automated tests on a daily basis. We are an ISTQB Platinum Partner and employ 100 testers certified at Advanced and Foundation levels. We carry out testing of mobile, web, and system applications.
