Depending on how continuous integration and regression testing is constructed in a project, there might be a need to run the same tests couple of times – without any changes to the test framework or tests suites itself. 

In such cases I often try to create a solution, which will not require rebuilding test framework every time I want to run my tests. This approach has couple of benefits, including the fact that it saves building time, but also, since we can build jar with all dependencies bundled, it provides the way to execute tests on machine with only JRE installed, which increases portability of our test framework. In this entry I would like to describe how do I build single jar test framework with cucumber tests and maven configuration. I’m assuming here that there is already created maven project with working cucumber tests.

Maven configuration

Creating runnable jar is quite standard I guess for all java applications. Personally, I use maven plugin maven-assembly-plugin to configure it from pom.xml level, so anytime I use command mvn clean package the jar is automatically created for me. The plugin uses descriptors to define what to put inside of the jar – there is possibility to use predefined descriptors (described here) and custom descriptor xmls. To build simplest jar with all dependencies it is enough to put following entry in your pom:

<plugin >
    <artifactId>maven-assembly-plugin</artifactId>
    <version>3.1.0</version>
    <configuration>
        <descriptorRefs>
            <descriptorRef>jar-with-dependencies</descriptorRef>
        </descriptorRefs>
    </configuration>
</plugin>

Descriptor jar-with-dependencies creates separate jar in target dir of the project – it contains all maven dependencies included in the configuration. In my case, however, it is often not enough.

Let’s assume I would like to include in my jar some files from outside of the projects home directory. In that case I need to create my own descriptor – it is done by referencing descriptor xml in plugin configuration:

    <configuration>
        ...
        <descriptors>
            <descriptor>src/assembly/descriptor.xml</descriptor>
        </descriptors>
    </configuration>

Additional descriptor xml needs to be created – it includes configuration specific for the jar to work properly. In below example I added single xml configuration file from outside of the project to the root of output jar, so it could be used in tests runtime:

<assembly xmlns="http://maven.apache.org/ASSEMBLY/2.0.0"
          xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
          xsi:schemaLocation="http://maven.apache.org/ASSEMBLY/2.0.0 http://maven.apache.org/xsd/assembly-2.0.0.xsd">
    <id>0.1</id>
    <formats>
        <format>jar</format>
    </formats>
    <includeBaseDirectory>false</includeBaseDirectory>
    <dependencySets>
        <dependencySet>
            <outputDirectory>/</outputDirectory>
            <useProjectArtifact>true</useProjectArtifact>
            <unpack>true</unpack>
            <scope>runtime</scope>
        </dependencySet>
    </dependencySets>
    <files>
        <file>
            <source>${basedir}/../configs/main_config.xml</source>
            <outputDirectory>/</outputDirectory>
        </file>
    </files>
</assembly>

One last thing to be configured in our plugin is reference to executable class with main method – it also goes into plugin configuration. Jar file, to be executable, needs to have main class referenced in MANIFEST file – it is automatically done for us by maven-assembly-plugin, when appropriately configured in pom:

    <configuration>
	 ...
        <archive>
            <manifest>
                <mainClass>
                    runners.TestRunner
                </mainClass>
            </manifest>
        </archive>
    </configuration>

After running mvn clean package, our plugin will build nice uber jar with our tests and all test dependencies. We did not yet mention anything about cucumber, because creating jar with dependencies does not depend on test libraries, so it can be done with any maven project out there. To be fully runnable, we still need to properly configure test runner class.

Executable cucumber test runner class

When using cucumber in classical way, there must be somewhere test runner class with appropriate CucumberOptions configured. Assuming we are using Junit:

@RunWith(Cucumber.class)
@CucumberOptions(
    features = "src/test/resources/feature ")

This is no longer the case, since Junit will not be running our tests anymore – we want to run it directly from command line, by using our assembled jar with executable class inside. Runner class must contain main method to be properly recognized by java as executable class, just like standard java applications. The simplest working implementation looks like in following example:

public final class TestRunner {
    public static void main(String[] args) {
        cucumber.api.cli.Main.main(args );
    }
}

Cucumber method cucumber.api.cli.Main.main(args) triggers whole cucumber engine and it takes parameters directly passed from our jars command line. There is a lot of parameters and they are quite powerful – information on how to use them can be found in here. So, at this point we have jar fully capable of running our tests:

java -jar cucumber-test-jar-with-dependencies.jar –glue cucumber/stepdefinition –plugin pretty classpath:features/MyTestFeature.feature

One thing to remember is that maven pom is no longer responsible for configuration and run of our tests – every configuration logic needs to be handled via test runner class. Ok, so far so good, but there is couple of things still to be done to make our jar more usable.

Command line parameters

There is no point in entering the same parameters every time – in my case step definitions glue parameter and plugins configurations are always identical and I wanted to hide them to not bring more confusion to jar usage than it is really required. To achieve that I simply hardcoded parameters, which are common for every test run – others, like feature files path and tags configuration can still be passed in command line:

public final class TestRunner {
    private static String[] defaultOptions = {
            "--glue", "stepdefinitions",
            "--plugin", "pretty",
            "--plugin", "json:cucumber.json"
    };
 
    public static void main(String[] args) {
        Stream<String> cucumberOptions = Stream.concat(Stream.of(defaultOptions), Stream.of(args));
        cucumber.api.cli.Main.main(cucumberOptions.toArray(String[]::new));
    }
}

But what about parameters, which previously were used in maven execution? In my case I usually configure my framework to take browser type as command line parameter – in case of maven it was passed to tests like that: mvn clean test -Dbrowser=chrome. Because this is handled as a System property in java code, in case of jar execution it still can be passed in command line, but slightly differently than cucumber parameters – it needs to be entered before -jar switch:

java -Dbrowser=chrome -jar cucumber-test-jar-with-dependencies.jar classpath:features/MyTestFeature.feature

Handling cucumber System.exit

There is an unusual behavior implemented inside of cucumber Main method – it calls System.exit at the end of test run. I’m not sure why it works like that – maybe just to throw specific error code when tests fail. It is troublesome though when using it as java api inside of our test runner class – It makes it impossible to put any logic after test execution as well as there is no possibility to ignore test failures – command execution will end as failure.

I found a way to ignore this by using java system manager. Among the others, it allows to listen for every call of System.exit inside of the application and inject some custom logic. I override checkExit(int status) method, so whenever exit is called I throw an exception from system manager, which then I am able to catch in my test runner. To make it work I also had to stub default checkPermission(Permission perm) implementation, because it threw some unwanted security exceptions:

public class IgnoreExitCall extends SecurityManager {
    @Override
    public void checkExit(int status) {
        throw new SecurityException();
    }

    @Override
    public void checkPermission(Permission perm) {
        //Allow other activities by default
    }
}

The last thing to do is to catch SecurityException in test runner class – it is unchecked exception, but we need to do that to prevent our application from exiting:

public final class TestRunner {
    public static void main(String[] args) {
        SecurityManager manager = new IgnoreExitCall();
        System.setSecurityManager(manager);
        try {
            cucumber.api.cli.Main.main(args);
        } catch (SecurityException) {
            System.out.println("Ignore exit");
        }
        //Do some other stuff like reporting logic
    }
}

That’s all! Happy testing!