Commonly Used Software Testing Frameworks and their Benefits - SQA Geek

Quality is never an accident;
it is always the result of intelligent effort.

Post Top Ad

Tuesday 12 May 2020

Commonly Used Software Testing Frameworks and their Benefits


What is a framework?

A framework is nothing but a set of protocols or rules that can be incorporated to leverage the benefits provided by the framework. A software testing framework provides an environment for the automation test scripts to be executed. With the use of framework, users can efficiently work with the automation test scripts, including development, execution, and reporting.

In short, a framework is a mixture of various guidelines, protocols, concept, processes, etc. that can be followed by a user while automating an application and take advantage of various positive aspects. Depending on the situation, the result could be in different forms, like scripting, modularity, scalability, maintenance, etc. These benefits could be grabbed if there are multiple test automation frameworks to be used for different software testing strategies.

However, there are situations when there is a requirement of a standard framework for test automation. This happens in a situation where there are multiple modules of an application and there are many developers who have their own idea of implementing automation. A single framework can help in avoiding any confusion arising due to multiple ideas.
Test automation frameworks by no means different. In general, a test automation framework is something that allows you to implement application-specific test automation solution by writing and injecting some additional code. Surprisingly, in test automation community by framework people often mean any test automation solution that happened to have any decent structure and design. Wikipedia as well supports this opinion [2].


framework
The most common “architecture type” of a test automation solutions I’ve seen was something that could be described as a “big ball of mud“. And, in some cases, it performs just great- such things don’t take ages to implement, quite easy to support for some shot period of time if you have limited amount of tests.
The really challenging question is why and when you need a real test automation framework and when you may go with plain “big ball of mud” test solution.

Start with question”why?”

All in all, test frameworks designed to address following issues:

1.Test complexity

The simpler the test on the top level is, the easier it is to create, understand, communicate and maintain.

2. Test maintenance cost

Test automation isn’t a thing that goes for granted, it has a price. One of the most painful prices for test automation is a maintenance cost. Well-designed framework should simplify maintenance.

3. Test execution time

Time is a money. Time to get a feedback from automated tests is also crucial – one wouldn’t like to wait for several hours just in order to check that his commit wasn’t wrong.

4. Reporting

Testing is about not finding bugs or accepting stories. Well, not exactly. What testing is really about – is providing an information. Test solution that has simple, maintainable tests, easy to maintain and enhance may be just thrown away if it produces shity reports that difficult to understand by non-technical stakeholders.
All things considered, if you don’t plan to have hundreds of complex tests scenarios, your tests are fast and you’re happy with built-in Xunit framework you may not need a test automation framework at all. The rule of thumb – the lower test level you’re interested in, the less complex should test automation solution be. In most cases, you don’t need a framework for unit or integration tests, however you may want one for acceptance tests.
Disregard of what test level you’re interested in, there’re some typical framework architectures that can be utilized. Most interestingly, framework architecture isn’t affected by test-automation approach applied (keyword-driven, BDD, plain code or whatever), because those approaches affects only some layers (most prominently test layers) and don’t touch others. Lets then take a look at the most know test framework architecture.
Layered architecture
While test automation framework are designed to address issues outlined above, it is usually isn’t overly complex software system. The most commonly known architecture pattern known as Layered Architecture is often used as a base for test automation framework architecture.
Components withing the layered architecture are organized into horizontal layers, and each layer of the architecture has specific role and responsibility in the application. Components within a specific layer are supposed to deal only with logic that corresponds to that layers [3].

Three-layered automation framework

The most known and widely used architecture for software test automation solutions is a three-layered architecture, in which solution is divided into three logical horizontal layers, usually test layer, business-layer and core layer.
Layered Test Automation Framework
In such architecture, test layer typically contains test scenarios itself, either in programming language or in any other form (like BDD feature files).
Business layer provides system under test (SuT) specific actions. For example, if we’re talking about online shopping, such actions may be log in, add something to cart, e.t.c.
Core layer is where real framework (who calls your code) really lives, and it deals with test orchestration, reporting and usually also provides low-level API to communicate with tested application, like web-service facades, Selenium Web Driver wrappers, e.t.c.
In a BDD inspired framework, test layer will typically contain feature files, business layer will contain steps definitions, while core layer contains BDD- framework configuration and core component. For a data-driven framework, test layer will contain data files and, business layer will contain mid-level application specific abstractions.

Four-Five layered automation framework

In case you may need something more complex, there are several options to enhance framework architecture. One of the options you may choose, is to extract validation code into separate level (or module, to be precise), which will be used by business layer, while core layer may provide interface or orchestration for validation.
In case you’re building hybrid framework, you may need separate layer for a data, so your test layer will have data and non-data tests. All this will add complexity to a core layer, so also logical may be to separate it into several sub-levels at some point.
four-five-layer

Plug-able test automation framework

In case you application is even more complex, you may want different validation for different environments, or different ways to communicate with application for same tests (for example for testing desktop and mobile versions of the web-page). While it is possible to address this by creating sub-modules in core or validation layers, there’s also neat way to do this completely opaque to the rest of the system. The idea is to use dependency injection and provide plug-able modules for validation and test-application interfacing.
plugable test automation framework
The idea is essentially the same as in previous framework architecture, with only difference – validation and facades implements provided interface, which is used by business layer and concrete implementation is typically injected without other layers knowledge about what was really injected.
Different Kinds of Test Automation Framework
There are different kind’s software testing methodologies and test automation framework available in the market. The wide range of framework available differs from each other on the basis of support and other key factors like, reusability, maintenance, etc.
Some of the most widely used test automation frameworks are discussed below.
  • Module based testing framework: Module based testing framework is based on the concept of abstraction one of the popular OOPS concept. Here the application is divided into a number of logical and isolated modules. An independent test script is created for each module. The modules are separated by an abstraction layer so that any change in the section doesn’t affect the module.

  • Library Architecture Testing Framework: This testing framework is actually built on the module based testing framework. However, this has some added benefits. Here, the application is divided into functions, rather than creating test scripts. The common functions can be used by other parts; hence, a common library is created with common functions for testing. The essence of the framework is to determine the similar steps and group them into functions, so that they can be called whenever required.

  • Data Driven Testing Framework: It is often required to test functionality multiple times with different set of input data. Here, instead of putting the test data in the test script, it is advised to keep the data in an external database. This will help in segregating the test script logic and test data from each other. This is exactly what test data driven framework does. The external database could be anything like XML, CSV, or ODBC repository. The data is generally kept in ‘Key-Value’ pair.

  • Keyword Driven Testing FrameworkThe keyword driven testing framework is built on the data driven testing framework, or we can say it is an extension of the same. It segregates the test data from scripts as well as certain code or keyword that is kept in an external file. These keywords are usually in the tabular form and are popularly known as a table driven framework as well.

  • Hybrid Testing Framework: Hybrid testing framework is a combination of more than one framework. This framework can get the advantage of all kinds of associated framework. Depending upon the situation, any of the above frameworks can be incorporated to build a hybrid framework.

  • Behavior Driven Development Framework: Behavior driven automation framework uses an easy understandable format that can be understood by analyst, developers, testers, etc. Such framework avoids the need to have knowledge of programming language by the user. There are various behavior driven tools available, including cucumber, Jbehave, etc.
The frameworks discussed above are some of the most commonly used frameworks used in the software testing industry. However, there are several other frameworks that can be used.

Considerations

While it is often tempting either to pick-up the most complex architecture (just in case) or the simplest thing (faster to implement), it is often wise to provide some initial investigation of what you may need based, for example, on product vision or expected length of life.
Provided architectures are by no means a standards, they are just examples, and you may implement plug-able three-layered thing, or even identify 7 different hierarchical layers. You may want some event-driven thing for testing of asynchronous workflow.
There are several typical highlights:
1)Try to make tests as simple, short and atomic as possible. Ideally, test should tell what is being done, not how it is done and test only one specific thing. Typically, cost of complex test support outweighs benefits of having such test.
2)Business layer should provide action implementation (how something is done) using interfaces provided by core layer
3)Validation layer should provide action implementation using interfaces provided either by core or business layer
4)Core layer is the framework itself, and may have complex architecture on its own. It’s responsibility is to provide interfaces for upper layer, provide orchestration for test run and reporting. It also often provides low-level implementation for interfaces exposed to upper layers.
All-in-all, there’s no right for all architecture, so you may suggest any other ways to separate test logic. Provided examples are just a place to start. Which is important, is that one should be able outline the architecture of the solution he is working on and guess if it fits to the project or application. My personal advice would be – stick to the simplest architecture that fits your immediate needs but try to leave options for you in case of need.

2 comments:

  1. Cool stuff you have and you keep overhaul every one of us esx script

    ReplyDelete
  2. Pretty good post. I just stumbled upon your blog and wanted to say that I have really enjoyed reading your blog posts. Any way I’ll be subscribing to your feed and I hope you post again soon. online crypter

    ReplyDelete

Post Bottom Ad