You may find the following useful if you are using a test framework such as Fitnesse where the developer/tester interacts with it using a web browser in order to create and run tests, and you want to make the framework available to developers without the inconvenience and inconsistency of local installs or creating a server specifically for it, and you also want to make it available to the CICD platform.
We got interested in this problem because we were starting another project using Fitnesse with DbFit and we wanted to standardise on the use of the framework so that we could focus fully on the actual testing work.
The requirements were:
The test framework must be available on the developer or tester’s laptop in an interactive form in order to develop tests
The test framework must be available to the CICD platform for running tests in a non interactive form
There must be no differences between the core functionality available from the test framework whether it is used by human users or by the CICD platform
Test definitions must be stored in the specific service repositories alongside the code that they test
Building and deploying a specific service should not require rebuilding the test framework
When improvements are made to the test framework they should be available to all services without effort to apply changes to each in turn
The solution concepts were
The test framework is defined in a standard test framework repository and is built as a standard test framework container image independently of the specific services that are tested with it
The standard test framework image can be run on the developer or tester’s laptop as follows
docker run the standard test framework
docker cp the current test definitions from the specific service repository to the test framework container
deploy the specific services to be tested either locally or in the cloud
edit the test definitions by interacting with the test framework
run the tests by interacting with the test framework, the test framework can use local or cloud resources
make changes to the service definitions and/or the tests until it is working as required
docker cp the updated test configuration back from the test framework container to the specific service repository
commit the test definitions along with any changes to the service definitions themselves
The same test framework image is also used by the CICD platform which copies the test definitions from the service definition repository into the standard test framework container and runs the tests in a non interactive form and collects the results.
Because the test framework standard image is built from the test framework repository, the individual service definition repositories do not need to contain the standard test framework components, and the tests run faster because the test framework components are not rebuilt
The solution in practice …
We created aliases for the docker cp command to copy test definitions to and from the framework container when editing tests. These turned out to be very quick and convenient. There are a number of more sophisticated ways to avoid the need to copy the edited test definitions back to the container for example using mounting of local directories, we may investigate those in future but at present they appear to add unnecessary complexity.
We created some standard test definitions in the test framework container itself as a way to demo the syntax and approach. Having these to refer to was helpful to avoid wasting time with the standard parts of test setup.
The specific service code itself may or may not have anything to do with building docker images. The system being tested could be local or in the cloud. In this example, we show it being used to test a simple serverless solution, where the specific service code is actually Terraform code which deploys the serverless solution into AWS:
Other Considerations and References…
This approach eliminated the need for separate test framework servers and also local installs and setting up an individual instance of the framework is now as simple as running the container with docker and using an alias to copy the test definitions to and from the relevant repository. When we invest effort in adding new features to the test framework image that make testing more efficient, we are then able to make use of them without additional overhead.