One of my highest-priority goals for the Muse Test Framework is extensibility. Initially, I made it easy to write new steps and value sources by simply implementing a few interfaces (in Java) and dropping a .jar file into the project folder. Then I developed an extension repository to distribute the extensions so users can easily install and update extensions in a project.
While working on some Muse extensions, I found that those features all worked very well and I was very pleased, But as work on those extensions continued, I found that more extensibility was needed. Most of the changes in 0.12 came about as I made more parts of the framework extensible…and also made the UI extensible in new ways.
As I worked on new capabilities for load testing extensions (to be announced soon), I found that I needed to change existing hard-coded meta-behaviors – that is, stuff that happens around the test execution, such as evaluating test pass/fail criteria, storing data produced by tests and executing a suite of tests. I initially tried to implement these using an existing hook – Context Initializers. These allowed access to the test execution context, but had only been designed for initializing data into the test context and the configuration capabilities were very limiting. The implementation proved insufficient for the task.
So Context Initializers needed to morph into Test Plugins. Just like the context initializers, a Test Plugin would be initialized into the test execution context and have access to context data as well as the events raised during test execution. Unlike context initializers, test plugins would be discovered dynamically (meaning they could be found in project extensions) and be easily configurable in the UI, just like any other project resource.
I immediately realized that many existing features would be better implemented as test plugins. The result is implementations that are less intertwined with the core framework, more configurable and most importantly – replaceable. If the implementation no longer suits your needs, you can easily code, install and configure a replacement into the project. These features are now implemented and configurable as plugins:
- Event Logger – records all events raised during the test
- Variable List Initializer – injects variables from a variable list into the test execution context (this was the initial context initializer)
- JUnit Report – generates a JUnit-compatible report from results collected during a test suite
- Local Storage Location – supplies a base location for storing results of tests and test suites
- Test Defaults Initializer – sets default variables defined in each test (on the Parameters tab) into the context before the test runs
- Test Result Calculator – determines success/failure/error status of each test by examining events raised during the test
- Test Suite Result Counter – summarizes the results of all tests in a test suite
- Test Results Saver – stores test results to disk as each test completes
- Compound Plugin – applies a group of plugins
Each of these plugins can be injected into tests (and test suites) based on configurable criteria. The screenshot below shows that I have added most of the new plugins to my project. The first (auto-plugins) is a List of Plugins plugin that contains all the others and applies them automatically to all tests and test suites.
Execution Context improvements
Tests have always executed within a TestExecutionContext provided by the framework. This context provides common services, such as storing/retrieving variables, accessing project resources and raising events. While this worked well enough, I found it needed to be extended up – to test suites and eventually to the entire project.
The 0.12 release includes the initial results of this work, starting with Test Suites. Each test in a suite still runs in a TestExecutionContext, but that now exists within a TestSuiteExecutionContext. This allows test suite plugins to look at all the tests together and interact with those tests within the larger context of the suite. As an example, the JUnit Report plugin can collect the data needed from each test, as it runs, and then create a test report to be stored when the test suite is complete. Similarly, the Test Results Saver plugin can ask each test plugin for it’s results when complete and save the results to disk…as well as looking for test suite results (such as a JUnit Report) and save it when the suite is complete.
Both individual tests and test suites execute within a ProjectExecutionContext. None of the plugins make use of this feature yet, but I expect to expose this in the Variable List Initializer plugin in the future. This will make it easier to initialize one variable list into anything that runs within project and an additional list into a specific test suite.
I found two deficiencies in the the event system:
- JSON output format was not sufficient for serialization and deserialization. It contained references to project assets, greatly increasing the storage size with duplicate data and complicating the deserialization process
- Extensions could not supply new kinds of events because EventType was an enumeration in the core library.
The event system was redesigned with a single event instance class that was flexible enough to hold the needed data while still being trivial to serialize and deserialize to JSON. A dynamic type system was added so extensions can now easily supply their own event types. A factory pattern using the type class to create event instances kept the code clean and readable (rather than abstracting away the important nuances of the code).
In this example (in Kotlin), line 9 (and 23) declares the new event type. The remainder (lines 11-21) is the factory method for creating an instance of the event, which stores the pass/fail state and the message as a tag and attribute in the event object.
Creating an instance remains simple and expressive:
This change did require some re-design and hard choices elsewhere in the core framework, but I am pleased with the end result…extensions can easily declare new event types and use them to expose new information during a test.
The Muse framework makes it very easy to implement new steps and value sources – and provides a friendly UI experience without writing any UI code. A few annotations on the implementation is sufficient for most steps and implementing a descriptor class covers the more challenging cases.
While this worked exactly as intended, it proved to be insufficient for the Page Objects extensions that I’ve been working on (to be announced soon). So the UI framework now provides extension points for providing custom implementations for rendering of steps or value source editors in the test building GUI.
As an example, the screenshot below shows a custom editor applied to the Click step by the Page Objects extension. Because this step is being edited within the context of a Start at Page step (its parent), a drop-down selection of available elements is shown instead of the usual text field for providing an element locator.
I have a hunch that much more will be needed along these lines, but the initial groundwork is in place and is ready to use by extension authors. Along the way, I’ve broken up the GUI project into several pieces in preparation for open-sourcing the code. I still have more work to do before I can do this with confidence, but look for more progress on this front in future releases 🙂
Extensible Test Suite Runner
While working on a parallel test runner for load testing (to be announced soon), I found that much of the code in the default (serial) runner and the GUI runner was not easily re-used and needed to be duplicated to build an extension. The base suite runner was re-designed to maximize re-use and allow the parallel runner to operate more consistently with the other runners. Runners can now be configured and specified on the command-line – so you can easily use the parallel runner or your own implementation when running in Jenkins or other CI environments.
Execution Id and Text Execution Id
Each execution within a project now receives a unique id. This feature is used by the Test Results Saver plugin to store test results to disk without over-writing previous test results.
New users should download the installer and get started! If you have already installed MuseIDE, you will be upgraded to 0.12 next time you start MuseIDE with internet access.