MuseIDE 0.12 release

One of my highest-priority goals for the Muse Test Framework is extensibility. Initially, I made it easy to write new steps and value sources by simply implementing a few interfaces (in Java) and dropping a .jar file into the project folder. Then I developed an extension repository to distribute the extensions so users can easily install and update extensions in a project.

While working on some Muse extensions, I found that those features all worked very well and I was very pleased, But as work on those extensions continued, I found that more extensibility was needed. Most of the changes in 0.12 came about as I made more parts of the framework extensible…and also made the UI extensible in new ways.

Test Plugins

As I worked on new capabilities for load testing extensions (to be announced soon), I found that I needed to change existing hard-coded meta-behaviors – that is, stuff that happens around the test execution, such as evaluating test pass/fail criteria, storing data produced by tests and executing a suite of tests. I initially tried to implement these using an existing hook – Context Initializers. These allowed access to the test execution context, but had only been designed for initializing data into the test context and the configuration capabilities were very limiting. The implementation proved insufficient for the task.

So Context Initializers needed to morph into Test Plugins. Just like the context initializers, a Test Plugin would be initialized into the test execution context and have access to context data as well as the events raised during test execution. Unlike context initializers, test plugins would be discovered dynamically (meaning they could be found in project extensions) and be easily configurable in the UI, just like any other project resource.

I immediately realized that many existing features would be better implemented as test plugins. The result is implementations that are less intertwined with the core framework, more configurable and most importantly – replaceable. If the implementation no longer suits your needs, you can easily code, install and configure a replacement into the project. These features are now implemented and configurable as plugins:

  • Event Logger – records all events raised during the test
  • Variable List Initializer – injects variables from a variable list into the test execution context (this was the initial context initializer)
  • JUnit Report – generates a JUnit-compatible report from results collected during a test suite
  • Local Storage Location – supplies a base location for storing results of tests and test suites
  • Test Defaults Initializer – sets default variables defined in each test (on the Parameters tab) into the context before the test runs
  • Test Result Calculator – determines success/failure/error status of each test by examining events raised during the test
  • Test Suite Result Counter – summarizes the results of all tests in a test suite
  • Test Results Saver – stores test results to disk as each test completes
  • Compound Plugin – applies a group of plugins

Each of these plugins can be injected into tests (and test suites) based on configurable criteria. The screenshot below shows that I have added most of the new plugins to my project. The first (auto-plugins) is a List of Plugins plugin that contains all the others and applies them automatically to all tests and test suites.

plugins

Execution Context improvements

Tests have always executed within a TestExecutionContext provided by the framework. This context provides common services, such as storing/retrieving variables, accessing project resources and raising events. While this worked well enough, I found it needed to be extended up – to test suites and eventually to the entire project.

The 0.12 release includes the initial results of this work, starting with Test Suites. Each test in a suite still runs in a TestExecutionContext, but that now exists within a TestSuiteExecutionContext. This allows test suite plugins to look at all the tests together and interact with those tests within the larger context of the suite. As an example, the JUnit Report plugin can collect the data needed from each test, as it runs, and then create a test report to be stored when the test suite is complete. Similarly, the Test Results Saver plugin can ask each test plugin for it’s results when complete and save the results to disk…as well as looking for test suite results (such as a JUnit Report) and save it when the suite is complete.

Both individual tests and test suites execute within a ProjectExecutionContext. None of the plugins make use of this feature yet, but I expect to expose this in the Variable List Initializer plugin in the future. This will make it easier to initialize one variable list into anything that runs within project and an additional list into a specific test suite.

Events

I found two deficiencies in the the event system:

  1. JSON output format was not sufficient for serialization and deserialization. It contained references to project assets, greatly increasing the storage size with duplicate data and complicating the deserialization process
  2. Extensions could not supply new kinds of events because EventType was an enumeration in the core library.

The event system was redesigned with a single event instance class that was flexible enough to hold the needed data while still being trivial to serialize and deserialize to JSON. A dynamic type system was added so extensions can now easily supply their own event types. A factory pattern using the type class to create event instances kept the code clean and readable (rather than abstracting away the important nuances of the code).

In this example (in Kotlin), line 9 (and 23) declares the new event type. The remainder (lines 11-21) is the factory method for creating an instance of the event, which stores the pass/fail state and the message as a tag and attribute in the event object.

GoalAssessmentEvent

Creating an instance remains simple and expressive:

GoalAssessmentEvent-create

This change did require some re-design and hard choices elsewhere in the core framework, but I am pleased with the end result…extensions can easily declare new event types and use them to expose new information during a test.

GUI extensibility

The Muse framework makes it very easy to implement new steps and value sources – and provides a friendly UI experience without writing any UI code. A few annotations on the implementation is sufficient for most steps and implementing a descriptor class covers the more challenging cases.

While this worked exactly as intended, it proved to be insufficient for the Page Objects extensions that I’ve been working on (to be announced soon). So the UI framework now provides extension points for providing custom implementations for rendering of steps or value source editors in the test building GUI.

As an example, the screenshot below shows a custom editor applied to the Click step by the Page Objects extension. Because this step is being edited within the context of a Start at Page step (its parent), a drop-down selection of available elements is shown instead of the usual text field for providing an element locator.

CustomInlineVSE

I have a hunch that much more will be needed along these lines, but the initial groundwork is in place and is ready to use by extension authors. Along the way, I’ve broken up the GUI project into several pieces in preparation for open-sourcing the code. I still have more work to do before I can do this with confidence, but look for more progress on this front in future releases 🙂

Extensible Test Suite Runner

While working on a parallel test runner for load testing (to be announced soon), I found that much of the code in the default (serial) runner and the GUI runner was not easily re-used and needed to be duplicated to build an extension. The base suite runner was re-designed to maximize re-use and allow the parallel runner to operate more consistently with the other runners. Runners can now be configured and specified on the command-line – so you can easily use the parallel runner or your own implementation when running in Jenkins or other CI environments.

Execution Id and Text Execution Id

Each execution within a project now receives a unique id. This feature is used by the Test Results Saver plugin to store test results to disk without over-writing previous test results.

Final notes

New users should download the installer and get started! If you have already installed MuseIDE, you will be upgraded to 0.12 next time you start MuseIDE with internet access.

MuseIDE 0.11 released

I’m happy to announce a new release with a few handy additions, some new extensibility built into the framework, some internal re-design of the GUI and the first 3rd-party project extension! The Muse Test Framework has been updated to version 0.11 as well.

UI for tags and metadata attributes

You can now add tags to test steps as well as metadata attributes (name/value pairs). The immediate application of this is the ability for other actors (e.g. context initializers) to customize their behavior based on the tags and attributes on a step, such as the Step Duration Goal Assessor in the measurement project extension (described below). Here is an example of a step with two tags (tag1 and tag2) and a metadata attribute (data1=value1).

TagsAndMetadata

Context Initializers in project extensions

When first implemented, a Context Initializer was only used to initialize variables in the test execution context at the start of a test. With version 0.11, the discovery, configuration and execution of context initializers has been re-designed so that they can be used to implement other capabilities, such as collecting or analyzing test events (such as computing step duration from start and end events). The IDE also has been updated to allow configuration of parameters required by the initializers.

The initializer group shown below has 3 initializers configured. The group has an application condition, that can be used to selectively apply (or not) the entire group. In this case it is true, meaning that it should always be applied. Each initializer also has an application condition. In the example, the first initializer is always applied and the second is applied only if the environment variable JOB_NAME is not defined. The result in this example is that if the tests are not running in Jenkins (which defines a JOB_NAME environment variable), the project-vars-local list will be applied, which may overwrite some of the variables defined in the project-vars list. In this case it allows the tests to run using a headless browser in Jenkins and a visible browser when running on the desktop.

ContextInitializersGroup

Data Collectors in project extensions

A new interface, DataCollector (and the related TestResultData), allows project extensions to collect data from a test. After the test, it may save that data with the other test artifacts. The event logger was re-implemented using these interfaces, so that the event log may now be saved when the test is run from the command line (using the -o option to specify an output location). A DataCollector is also a Context Initializer, so it can be configured in the same way as the initializers described above.

As an example, here is the output of a step duration collector (described later).

StepDurationsJson

New Tree in the Step Test editor

The tree control in the test test editor, which displays the steps, has been re-written from scratch. This component was written before I discovered the TestFX library for unit-testing JavaFX. As a result, it had no unit tests…not a good thing for the most complicated part of the GUI. It now has a decent set unit tests. Still not as comprehensive as I’d like, but much better. The functionality should remain the same with a few minor improvements. But I’m sure I missed a few defects, so please report bugs as you find them.

Measurements extension

Web Performance has sponsored development of the first 3rd-party project extension for Muse. The extension is for measuring the duration of steps in tests. It provides two new capabilities.

The first is called the Step Duration Collector. It will record (and store) the duration of all steps in the test (for later analysis). Earlier on this page is an example of the output of the collector. Each step is identified by its ID, which allows analysis code to refer back to the test definition for details on the step. Since a step can run more than once in a test, each step has an array of durations (in milliseconds).

The second new capability, the Step Duration Goal Assessor, compares the duration of a step to a configured goal. When a step exceeds the configured duration goal, a failure event is raised (which will cause the test to be marked as a failure).

FailedGoalEvent

As with other extensions, it is available via the Extensions… button. Read more about the measurements extension.

MuseIDE 0.10 released

The 0.10 release brings the ability to add extensions to your project. The currently available extensions make it easy to install the most popular drivers for Selenium WebDriver – ChromeDriver, GeckoDriver and Microsoft Web Driver.

manage-extensions-pane

Press the Extensions… button to open the Extensions Manager (1). Then locate the extension you want and press the ‘+‘ button (2). Be sure the watch the install log that appears – you want to see “Installation successful” at the end! (3)

To remove an extension, switch to the Installed tab and press the ‘‘ button to remove it.

Note that after downloading the drivers, you’ll still need to configure the Browser and Browser Provider as shown in the Setting up a project tutorial.

MuseIDE 0.9 released

You can now run your test suites within MuseIDE, watch the progress, cancel when desired and view the result and event log for each test in the suite. See the Run Test Suites in MuseIDE post for details.

run-suite

Other enhancements include:

  • IdListTestSuite editor supports adding ids of test suites…allowing suites containing other suites
  • enhance expression grammar for simpler usage: <page.element> instead of <page:”page.element”>
  • improved docking behavior (no floating sections, navigator not maximizable)
  • When starting a test suite, start running immediately (don’t make user press 2 play buttons)
  • StepTree is smarter about auto-collapsing only the branches that it auto-expanded when running a test interactively
  • Support for the PhantomJS driver
  • new step: SetBrowserSize
  • new condition: ElementHasClassCondition (and expression support)
  • Improvements to test execution support classes to improve the test suite reports
  • Improvements to the JUnitReportRenderer
  • Update to Selenium 3.4

Bugs fixed:

  • After deleting a step, undoing, redoing and undoing again results in 2 copies of the un-deleted step.
  • Running a test in one editor results in the other test editors acting like a test is running.
  • When importing SelniumIDE tests, if the last-used folder no longer exists, the select-files dialog does not open.
  • Ignore exceptions thrown by WebDriver.quit()
  • Tests always return a result, even when serious exceptions are thrown
  • encode jUnit report XML in UTF-8 (for Jenkins)
  • HTML-escape the test failure message in the jUnit report XML
  • run VariableListContextInitializers in the order they appear in the config

Run Test Suites in MuseIDE

Typically, test suites are designed to be run via the command line interface in a scheduled job or a CI server such as Jenkins or Travis CI. The Muse Test Framework has long supported this capability. Now, a test suite can be run in the MuseIDE, too!

Start by pressing the Run button at the top of the test suite editor. This will open the suite progress panel. Press the Run button in the panel to start the suite:

run-suite

After the suite has completed, press the Details button to view the status and event log for each test:

suite-results

Note: this feature first appears in MuseIDE version 0.9.

Import SeleniumIDE tests into MuseIDE

MuseIDE for Selenium supports supports the following commands, targets, matchers when importing SeleniumIDE tests:

Commands

  • click, clickAndWait
  • open
  • goBack, goBackAndWait
  • refresh, refreshAndWait
  • pause (becomes Wait)
  • select (label= and index=)
  • type, sendKeys
  • store
  • verifyElementPresent, assertElementPresent
  • verifyText, assertText
  • verify, assert

Element Targets

  • css
  • id
  • name
  • link
  • xpath

Parameters

  • javascript{…}
  • ${varname}
  • exact:
  • glob:
  • regexp:
  • regexpi:

Note: This list applies to MuseIDE version 0.8.

How to: Import SeleniumIDE tests into MuseIDE

MuseIDE can import SeleniumIDE tests via the Import button in the Project Navigator. Many files can be imported at once; this UI shows some of the problems that can occur during the import process:

seleniumide-importui

A status is shown for each test to be imported. The 4 tests in the above example illustrate:

  1. will import correctly
  2. could not be read and parsed – it might just be a regular HTML file
  3. will be imported, but uses some commands that are not recognized – some manual adjustments will be needed
  4. could be imported if the user selects the Import? checkbox – but it is initially disabled because it would overwrite an existing test in the project

If you would like the original source files to be deleted, check the Delete source files… option before starting the import.

Note: this feature appears in MuseIDE version 0.8.