I’m happy to announce a new release with a few handy additions, some new extensibility built into the framework, some internal re-design of the GUI and the first 3rd-party project extension! The Muse Test Framework has been updated to version 0.11 as well.
UI for tags and metadata attributes
You can now add tags to test steps as well as metadata attributes (name/value pairs). The immediate application of this is the ability for other actors (e.g. context initializers) to customize their behavior based on the tags and attributes on a step, such as the Step Duration Goal Assessor in the measurement project extension (described below). Here is an example of a step with two tags (tag1 and tag2) and a metadata attribute (data1=value1).
Context Initializers in project extensions
When first implemented, a Context Initializer was only used to initialize variables in the test execution context at the start of a test. With version 0.11, the discovery, configuration and execution of context initializers has been re-designed so that they can be used to implement other capabilities, such as collecting or analyzing test events (such as computing step duration from start and end events). The IDE also has been updated to allow configuration of parameters required by the initializers.
The initializer group shown below has 3 initializers configured. The group has an application condition, that can be used to selectively apply (or not) the entire group. In this case it is true, meaning that it should always be applied. Each initializer also has an application condition. In the example, the first initializer is always applied and the second is applied only if the environment variable JOB_NAME is not defined. The result in this example is that if the tests are not running in Jenkins (which defines a JOB_NAME environment variable), the project-vars-local list will be applied, which may overwrite some of the variables defined in the project-vars list. In this case it allows the tests to run using a headless browser in Jenkins and a visible browser when running on the desktop.
Data Collectors in project extensions
A new interface, DataCollector (and the related TestResultData), allows project extensions to collect data from a test. After the test, it may save that data with the other test artifacts. The event logger was re-implemented using these interfaces, so that the event log may now be saved when the test is run from the command line (using the -o option to specify an output location). A DataCollector is also a Context Initializer, so it can be configured in the same way as the initializers described above.
As an example, here is the output of a step duration collector (described later).
New Tree in the Step Test editor
The tree control in the test test editor, which displays the steps, has been re-written from scratch. This component was written before I discovered the TestFX library for unit-testing JavaFX. As a result, it had no unit tests…not a good thing for the most complicated part of the GUI. It now has a decent set unit tests. Still not as comprehensive as I’d like, but much better. The functionality should remain the same with a few minor improvements. But I’m sure I missed a few defects, so please report bugs as you find them.
Web Performance has sponsored development of the first 3rd-party project extension for Muse. The extension is for measuring the duration of steps in tests. It provides two new capabilities.
The first is called the Step Duration Collector. It will record (and store) the duration of all steps in the test (for later analysis). Earlier on this page is an example of the output of the collector. Each step is identified by its ID, which allows analysis code to refer back to the test definition for details on the step. Since a step can run more than once in a test, each step has an array of durations (in milliseconds).
The second new capability, the Step Duration Goal Assessor, compares the duration of a step to a configured goal. When a step exceeds the configured duration goal, a failure event is raised (which will cause the test to be marked as a failure).
As with other extensions, it is available via the Extensions… button. Read more about the measurements extension.