Introduction

Nowadays test automation is an integral part of software development process. I decided to create series of blog entries, which will describe how to create customized Selenium test framework in java from scratch to make it fully functional, easy to use and maintain. After years of creating various test frameworks in number of projects I can say that before you start it is important to ask yourself some basic questions. Answers to those questions may surprisingly redefine test automation architecture or even whole testing approach in the project. There are no simple, universal answers, but from my experience it is beneficial to think about them at the beginning.

Why to automate at all?

It is always good thing to spend some time on trying to understand the reason why do you actually need to create automated tests in your project. Good automation usually comes from real needs, not from management or hope for a better future. You can find a lot of information about benefits of test automation – how it can save time and money, increase test coverage, how it is better than manual testing in general. But when implementing functional tests in java with Selenium there are some drawbacks, which you need to understand and be prepared to face:

• Functional test automation is often more expensive than manual testing at least at implementation stage. It will take time to create framework, to implement tests and later on to maintain those tests due to their instability in the future. It may happen that manual testers will be able to test large part of product, while you will be firefighting dozens of configuration and dependency issues in test environment to run your framework.
• In most cases it won’t be as good as manual tests in finding defects in newly introduced product features. To create automated test you need to go through test scenario manually anyway.
• Functional tests based on Selenium tend to be unstable and usually requires constant attention. This is because end to end tests depend on many parts of the system, which change over time.
• Test framework is a software project. It needs to have own architecture, documentation, budget and finally engineers with coding skills.

Selenium is used to automate communication with web browser and when combined with java it is mostly used to implement functional end to end tests. As for my experience, they are perfect to create regression suites, which will verify if nothing was broken when introducing new features to the product.

Who is going to use the framework?

When creating a software product we usually think about end users. A lot of effort is put to ensure that application is easy to use, stable and free from critical defects. As I mentioned before, test framework can be treated as a software project as well – in such case, who should we consider as end users? Here is my list of customers I think is important to remember about:

• Management – framework needs to have presentation layer, which would be readable for non-technical person to understand what is tested. Jenkins job status is definitely not enough – perfectly when each test scenario is described in human language and has own descriptive failure description.
• Developers – on some level it also needs to provide as much debug information as possible. Screenshots, videos, test scenario logs, test environment logs and anything more that would be helpful in understanding what happened without the need of rerunning test again – it’s not always possible to reproduce the problem second time.
• Continuous integration – end to end tests are slow comparing to unit or integration tests. Some of them can be used for development commits verification – others can be run on demand or periodically to verify current status of the product. Keep that in mind when planning test suites and way of returning final test result.
• Test developers – last but not least. Even if you’re the only one who creates and maintain testing solution – this may change in the future. Framework architecture needs to be clearly described and documented. Testing API needs to be as much self-explanatory as possible to make test methods easy to understand and implement. All coding principles apply.

What is to be tested?

One of the most important things when designing test framework architecture is to define what is to be tested. In my humble opinion this is the most critical part, which decides if your test approach will help your team to deliver or will be rejected from the process. Clear distinction between e2e, integration and unit as well as between functional and non-functional testing is essential.

You could expect that everyone understands by now what test pyramid is and how it works. While unit tests are cheapest and fastest, they should cover as much code as possible. Integration tests are slower, but still way faster than functional end to end tests. Surprisingly often this knowledge is somehow forgotten and not taken into consideration when software project grows and new functionalities need to be tested. Yes, Selenium can be used to gain unit coverage – Protractor, Nightwatch or similar frameworks would be way to go while unit testing UI component – in separation from the system. Also it can be used on integration testing level, but again when UI layer is designed as a separate component to the rest of the system. Similarly it is not good idea to use Selenium functional test framework for performance testing (sounds abstract, but it really happens). My point is that when you think about java based Selenium framework to test software product, which need all its components to be running – those are the slowest, most expensive and most fragile tests to run.

Remember that when there is a need to test new functionality following tests need to be created:

• Unit tests – focused on testing product code with best coverage, run every build – while created by developers, in my opinion should also be reviewed by QA engineers. It gives understanding how the product is tested on low level and how it can be complemented by integration and e2e tests. 
• Integration tests – cover all variations, combinations, corner cases imaginable. They should replace as much cases from e2e test layer as possible. Ideally they could be included in building process of every product component. Effort to create those tests can be distributed between DEV and QA engineers and everyone must understand scope and coverage of those tests.
• End to end tests – those tests focus on verifying if given functionality of the system works in general. They should clearly describe what is tested – all functionalities need to be covered, but never ever try to achieve any code coverage on this level. Some corner cases could be covered here, depending on complexity of the system under test, but I would recommend pushing all variations to integration tests.

Everything has its own purpose and this also applies to testing in software development. You can always reuse some parts of test framework for different types of testing, but this must be designed thoroughly from the beginning.

What knowledge is needed?

Object oriented programming

If you plan to create complete, stable and reusable testing solution in java to test functionalities of the product, then you need to have or gain knowledge about object programming principles and good practices. What’s more, you will need to create your own rules and good practices for others, whoever would like to contribute to the framework. From my experience good design of the framework internals automatically brings clarity on test method level. In following articles I will try to explain how this can be achieved in details, but in general you should be familiar with all SOLID principles and basic design patterns.

Selenium

Selenium came long way from its first versions to what it is now. Old Selenium-RC was completely rewritten to WebDriver approach, which brought browser’s built-in support, great testing API and simplified test architecture. While originally it’s Selenium that was responsible for providing drivers for communication with browsers, now, thanks to W3C specification all major browser vendors delivers their own WebDriver implementations.

From java perspective WebDriver API is intuitive and easy to use but to make functional test methods meaningful and concise do not use it directly in tests. To handle this you could use dedicated Page Object design pattern followed by Page Object Factory.

You should also avoid java test code generators at all costs. Generated code provides easiest solutions from coding perspective in opposite to what we want to achieve by building test framework – reusable, easy to use and understand solution.

Java builders and test runners

When building testing solution for a software project, it feels like natural thing to reuse its configuration in test framework as well. It might not always be the best though. Try to familiarize yourself with key concepts of JUnit and TestNG as well as Maven and Gradle to pick what will best fit your needs.

What’s next?

Now, when you exactly now what you need from product testing perspective let’s make it work. In next part I will explain how to build nice, maintainable and easy to use test framework.