MuseIDE 0.13 release

In short:

Browsers can now be run in headless mode via the arguments in the browser provider configuration. Tests recorded or built with the new SeleniumIDE (3.x) can now be imported into MuseIDE from the .side file.

In detail:

Additions and improvements

  • Import new SeleniumIDE format files (.side)
  • BrowserProvider now has an arguments field that is passed to the browser startup command. E.g. “–headless”
  • ValueSourceDescriptors can now have a default value. This is useful, for example, on optional step parameters that should have a ‘true’ value when used.
  • All events are shown in the event log table, including those before the first step starts.
  • Updated to Selenium 3.11
  • Improve command-line output from tests
  • New WebdriverCapture plugin can collect screenshots, page content and log files for each step.
  • New EventLogWriter plugin writes events to disk as-they-are-received, instead of at the end of the test. Very useful when a browser hangs or test crashes due to unhandled errors.

Bug fixes

  • Fixed bug with copy and multiple-paste that caused step-ids to be duplicated, leading to UI corruption when stepping through a test.
  • Fix: Test Suite plugin failures do not show message to user (command-line or log file).
Advertisements

SeleniumIDE 3.x File Import (.side files)

The 0.13 release (coming soon) adds the ability to import tests in .side files – the format created by the newest SeleniumIDE (3.x). This is an enhancement to the existing import, which could import the previous SeleniumIDE format (.html).

To import SeleniumIDE tests into your project:

  1. Press the Import button in the project navigator
  2. Select the .side file
  3. Select the tests to import
  4. Press the Import N tests button

When selecting tests to import, the test name (assigned in SeleniumIDE) will be converted into a filename-friendly resource id. The new id is shown on the import screen and id conflicts will be indicated:

Import-duplicateId

When selecting the tests to import, also you can see if tests have any unrecognized commands:

Import-unrecognizedCommands

Tests with unsupported commands can still be imported – they will be noted in the step editor in MuseIDE along with their command names and parameters. You can then decide if these steps are necessary (SeleniumIDE records a lot of unnecessary steps). If you find there are critical commands that are unrecognized, please contact us. Frequently, these commands are easy to add and we can issue an update within a day or two.

The test editor indicates steps that use unrecognized commands with a red warning icon:

Import-unrecognizedCommandsInEditor

In this case, the mouse and scroll steps are unnecessary. Removing those steps results in a test that executes successfully. Your mileage may vary.

The ability to import .side files means you can record a workflow in SeleniumIDE and then quickly import into a MuseIDE test, rather than building the tests step-by-step. SeleniumIDE is available as an extension for Firefox and Chrome.

 

PageObjects extension for MuseIDE

One of my primary goals for the design of MuseIDE and the support for building Selenium-based tests is to lower the cost of building and maintaining tests built in a visual IDE.

There is a design pattern that test automation engineers use to reduce maintenance cost in coded tests: the Page Objects pattern. The purpose of the pattern is to separate the details of the page implementation (such as how to locate an element on a page) from the logic of the tests. In doing so, and by encapsulating the page implementation details into classes, duplication of those details is greatly reduced…or even eliminated entirely. In turn, this reduces maintenance cost because when those details change, they only need to be updated in one place. It can also reduce the cost of test development by building logic that can be re-used easily in future tests. Here is a more detailed code example of Page Objects in coded tests if you are curious to see what test code may look like.

When you look at the classes built for a Page Objects implementation, you are essentially seeing a model of the application – the pages and the elements, data sources and actions that are contained on each page. You may also notice that the things that are modeled (pages, elements, data sources, actions) will be very similar for each web application tested…even for applications that look very different on the surface. This implies an opportunity to build a general-purpose model and GUI that will serve for many or most applications.

Opportunities for re-use

Lets start by looking at the MuseIDE Selenium test shown below — it performs a few simple steps:

  1. Start at a URL
  2. Click a link to the login page
  3. Login (fill username & password, press login button)
  4. Verify the login succeeded
  5. If the logout button is present, press it, wait for completion and check for success
  6. Close the browser

login-logout-test-without-POs

The login process has been implemented as a reusable function, so that is good. But nearly every other step has details that are likely to be duplicated in other tests that deal with these pages. Some examples

  1. “Go to…” step has a hard-coded URL. Every test that starts on this page will have a copy…and they would all need to change if that URL changes.
  2. Each of the click steps contains an element locator, which may be duplicated in other tests.
  3. Each of the verify steps references a condition on the page, which is duplicated in the login function and may be duplicated in other tests as well.

Convert to use Page Objects

Here is the same test, converted to use the Page Objects extension. You will immediately notice two differences: (1) the test is much more expressive – anyone could read it and immediately understand what each step in test is doing – and (2) all of the nitty-gritty details of the implementation have vanished. You may also notice a 3rd difference – there is a little more hierarchy in the test. The steps are separated into those on the HomePage and those on the LoginPage.

login-logout-using-POs

So where did all the details go? They are moved into Web Pages defined in the project. The first, the HomePage, is very simple in this example, so lets look at it first.

HomePage

The HomePage only has two components defined – the Page URL and one element – the LoginPageLink.

HomePage-PO

When the Start at Page step executes, it looks up the URL for the page and sends the URL to the browser. Each test that starts at the HomePage no longer duplicates the URL – it is defined in this one place.

One element has been defined on the page – the LoginPageLink. The element locator has been moved here and is referenced by name in the click step of the test.

LoginPage

The LoginPage has many more elements defined, as well an action and a few conditions.

LoginPage-PO

The Perform Action step in the test references an Action on the page. The action simply references a function defined in the project.

The Verify and Wait steps in the test reference conditions on the page. Again, the details have been moved out of the test and into the page model.

Also notice that the conditions do not duplicate the locators for the elements they reference – they use a special notation to reference the element as defined on the page.

Login function

The login function was also converted to use Page Objects. Here is the initial version:

LoginFunction-without-POs

And here is the Page Objects version:

LoginFunction-using-POs.png

Converting the test to model the HomePage and LoginPage in this way has eliminated duplication of the element locators and conditions, as well as making the tests easier to read. When minor changes are made to the application that affect element locators, only the Page Objects will need updates. Each change will be made in just one place – reducing cycle time and lowering costs.

To get MuseIDE, download the installer.

To install the Page Objects extension into your project, press the Extensions… button in the project navigator, switch to the Available tab and press the + button next to the Page Objects extension.

MuseIDE 0.12 release

One of my highest-priority goals for the Muse Test Framework is extensibility. Initially, I made it easy to write new steps and value sources by simply implementing a few interfaces (in Java) and dropping a .jar file into the project folder. Then I developed an extension repository to distribute the extensions so users can easily install and update extensions in a project.

While working on some Muse extensions, I found that those features all worked very well and I was very pleased, But as work on those extensions continued, I found that more extensibility was needed. Most of the changes in 0.12 came about as I made more parts of the framework extensible…and also made the UI extensible in new ways.

Test Plugins

As I worked on new capabilities for load testing extensions (to be announced soon), I found that I needed to change existing hard-coded meta-behaviors – that is, stuff that happens around the test execution, such as evaluating test pass/fail criteria, storing data produced by tests and executing a suite of tests. I initially tried to implement these using an existing hook – Context Initializers. These allowed access to the test execution context, but had only been designed for initializing data into the test context and the configuration capabilities were very limiting. The implementation proved insufficient for the task.

So Context Initializers needed to morph into Test Plugins. Just like the context initializers, a Test Plugin would be initialized into the test execution context and have access to context data as well as the events raised during test execution. Unlike context initializers, test plugins would be discovered dynamically (meaning they could be found in project extensions) and be easily configurable in the UI, just like any other project resource.

I immediately realized that many existing features would be better implemented as test plugins. The result is implementations that are less intertwined with the core framework, more configurable and most importantly – replaceable. If the implementation no longer suits your needs, you can easily code, install and configure a replacement into the project. These features are now implemented and configurable as plugins:

  • Event Logger – records all events raised during the test
  • Variable List Initializer – injects variables from a variable list into the test execution context (this was the initial context initializer)
  • JUnit Report – generates a JUnit-compatible report from results collected during a test suite
  • Local Storage Location – supplies a base location for storing results of tests and test suites
  • Test Defaults Initializer – sets default variables defined in each test (on the Parameters tab) into the context before the test runs
  • Test Result Calculator – determines success/failure/error status of each test by examining events raised during the test
  • Test Suite Result Counter – summarizes the results of all tests in a test suite
  • Test Results Saver – stores test results to disk as each test completes
  • Compound Plugin – applies a group of plugins

Each of these plugins can be injected into tests (and test suites) based on configurable criteria. The screenshot below shows that I have added most of the new plugins to my project. The first (auto-plugins) is a List of Plugins plugin that contains all the others and applies them automatically to all tests and test suites.

plugins

Execution Context improvements

Tests have always executed within a TestExecutionContext provided by the framework. This context provides common services, such as storing/retrieving variables, accessing project resources and raising events. While this worked well enough, I found it needed to be extended up – to test suites and eventually to the entire project.

The 0.12 release includes the initial results of this work, starting with Test Suites. Each test in a suite still runs in a TestExecutionContext, but that now exists within a TestSuiteExecutionContext. This allows test suite plugins to look at all the tests together and interact with those tests within the larger context of the suite. As an example, the JUnit Report plugin can collect the data needed from each test, as it runs, and then create a test report to be stored when the test suite is complete. Similarly, the Test Results Saver plugin can ask each test plugin for it’s results when complete and save the results to disk…as well as looking for test suite results (such as a JUnit Report) and save it when the suite is complete.

Both individual tests and test suites execute within a ProjectExecutionContext. None of the plugins make use of this feature yet, but I expect to expose this in the Variable List Initializer plugin in the future. This will make it easier to initialize one variable list into anything that runs within project and an additional list into a specific test suite.

Events

I found two deficiencies in the the event system:

  1. JSON output format was not sufficient for serialization and deserialization. It contained references to project assets, greatly increasing the storage size with duplicate data and complicating the deserialization process
  2. Extensions could not supply new kinds of events because EventType was an enumeration in the core library.

The event system was redesigned with a single event instance class that was flexible enough to hold the needed data while still being trivial to serialize and deserialize to JSON. A dynamic type system was added so extensions can now easily supply their own event types. A factory pattern using the type class to create event instances kept the code clean and readable (rather than abstracting away the important nuances of the code).

In this example (in Kotlin), line 9 (and 23) declares the new event type. The remainder (lines 11-21) is the factory method for creating an instance of the event, which stores the pass/fail state and the message as a tag and attribute in the event object.

GoalAssessmentEvent

Creating an instance remains simple and expressive:

GoalAssessmentEvent-create

This change did require some re-design and hard choices elsewhere in the core framework, but I am pleased with the end result…extensions can easily declare new event types and use them to expose new information during a test.

GUI extensibility

The Muse framework makes it very easy to implement new steps and value sources – and provides a friendly UI experience without writing any UI code. A few annotations on the implementation is sufficient for most steps and implementing a descriptor class covers the more challenging cases.

While this worked exactly as intended, it proved to be insufficient for the Page Objects extensions that I’ve been working on (to be announced soon). So the UI framework now provides extension points for providing custom implementations for rendering of steps or value source editors in the test building GUI.

As an example, the screenshot below shows a custom editor applied to the Click step by the Page Objects extension. Because this step is being edited within the context of a Start at Page step (its parent), a drop-down selection of available elements is shown instead of the usual text field for providing an element locator.

CustomInlineVSE

I have a hunch that much more will be needed along these lines, but the initial groundwork is in place and is ready to use by extension authors. Along the way, I’ve broken up the GUI project into several pieces in preparation for open-sourcing the code. I still have more work to do before I can do this with confidence, but look for more progress on this front in future releases 🙂

Extensible Test Suite Runner

While working on a parallel test runner for load testing (to be announced soon), I found that much of the code in the default (serial) runner and the GUI runner was not easily re-used and needed to be duplicated to build an extension. The base suite runner was re-designed to maximize re-use and allow the parallel runner to operate more consistently with the other runners. Runners can now be configured and specified on the command-line – so you can easily use the parallel runner or your own implementation when running in Jenkins or other CI environments.

Execution Id and Text Execution Id

Each execution within a project now receives a unique id. This feature is used by the Test Results Saver plugin to store test results to disk without over-writing previous test results.

Final notes

New users should download the installer and get started! If you have already installed MuseIDE, you will be upgraded to 0.12 next time you start MuseIDE with internet access.

MuseIDE 0.11 released

I’m happy to announce a new release with a few handy additions, some new extensibility built into the framework, some internal re-design of the GUI and the first 3rd-party project extension! The Muse Test Framework has been updated to version 0.11 as well.

UI for tags and metadata attributes

You can now add tags to test steps as well as metadata attributes (name/value pairs). The immediate application of this is the ability for other actors (e.g. context initializers) to customize their behavior based on the tags and attributes on a step, such as the Step Duration Goal Assessor in the measurement project extension (described below). Here is an example of a step with two tags (tag1 and tag2) and a metadata attribute (data1=value1).

TagsAndMetadata

Context Initializers in project extensions

When first implemented, a Context Initializer was only used to initialize variables in the test execution context at the start of a test. With version 0.11, the discovery, configuration and execution of context initializers has been re-designed so that they can be used to implement other capabilities, such as collecting or analyzing test events (such as computing step duration from start and end events). The IDE also has been updated to allow configuration of parameters required by the initializers.

The initializer group shown below has 3 initializers configured. The group has an application condition, that can be used to selectively apply (or not) the entire group. In this case it is true, meaning that it should always be applied. Each initializer also has an application condition. In the example, the first initializer is always applied and the second is applied only if the environment variable JOB_NAME is not defined. The result in this example is that if the tests are not running in Jenkins (which defines a JOB_NAME environment variable), the project-vars-local list will be applied, which may overwrite some of the variables defined in the project-vars list. In this case it allows the tests to run using a headless browser in Jenkins and a visible browser when running on the desktop.

ContextInitializersGroup

Data Collectors in project extensions

A new interface, DataCollector (and the related TestResultData), allows project extensions to collect data from a test. After the test, it may save that data with the other test artifacts. The event logger was re-implemented using these interfaces, so that the event log may now be saved when the test is run from the command line (using the -o option to specify an output location). A DataCollector is also a Context Initializer, so it can be configured in the same way as the initializers described above.

As an example, here is the output of a step duration collector (described later).

StepDurationsJson

New Tree in the Step Test editor

The tree control in the test test editor, which displays the steps, has been re-written from scratch. This component was written before I discovered the TestFX library for unit-testing JavaFX. As a result, it had no unit tests…not a good thing for the most complicated part of the GUI. It now has a decent set unit tests. Still not as comprehensive as I’d like, but much better. The functionality should remain the same with a few minor improvements. But I’m sure I missed a few defects, so please report bugs as you find them.

Measurements extension

Web Performance has sponsored development of the first 3rd-party project extension for Muse. The extension is for measuring the duration of steps in tests. It provides two new capabilities.

The first is called the Step Duration Collector. It will record (and store) the duration of all steps in the test (for later analysis). Earlier on this page is an example of the output of the collector. Each step is identified by its ID, which allows analysis code to refer back to the test definition for details on the step. Since a step can run more than once in a test, each step has an array of durations (in milliseconds).

The second new capability, the Step Duration Goal Assessor, compares the duration of a step to a configured goal. When a step exceeds the configured duration goal, a failure event is raised (which will cause the test to be marked as a failure).

FailedGoalEvent

As with other extensions, it is available via the Extensions… button. Read more about the measurements extension.

MuseIDE 0.9 released

You can now run your test suites within MuseIDE, watch the progress, cancel when desired and view the result and event log for each test in the suite. See the Run Test Suites in MuseIDE post for details.

run-suite

Other enhancements include:

  • IdListTestSuite editor supports adding ids of test suites…allowing suites containing other suites
  • enhance expression grammar for simpler usage: <page.element> instead of <page:”page.element”>
  • improved docking behavior (no floating sections, navigator not maximizable)
  • When starting a test suite, start running immediately (don’t make user press 2 play buttons)
  • StepTree is smarter about auto-collapsing only the branches that it auto-expanded when running a test interactively
  • Support for the PhantomJS driver
  • new step: SetBrowserSize
  • new condition: ElementHasClassCondition (and expression support)
  • Improvements to test execution support classes to improve the test suite reports
  • Improvements to the JUnitReportRenderer
  • Update to Selenium 3.4

Bugs fixed:

  • After deleting a step, undoing, redoing and undoing again results in 2 copies of the un-deleted step.
  • Running a test in one editor results in the other test editors acting like a test is running.
  • When importing SelniumIDE tests, if the last-used folder no longer exists, the select-files dialog does not open.
  • Ignore exceptions thrown by WebDriver.quit()
  • Tests always return a result, even when serious exceptions are thrown
  • encode jUnit report XML in UTF-8 (for Jenkins)
  • HTML-escape the test failure message in the jUnit report XML
  • run VariableListContextInitializers in the order they appear in the config

Run Test Suites in MuseIDE

Typically, test suites are designed to be run via the command line interface in a scheduled job or a CI server such as Jenkins or Travis CI. The Muse Test Framework has long supported this capability. Now, a test suite can be run in the MuseIDE, too!

Start by pressing the Run button at the top of the test suite editor. This will open the suite progress panel. Press the Run button in the panel to start the suite:

run-suite

After the suite has completed, press the Details button to view the status and event log for each test:

suite-results

Note: this feature first appears in MuseIDE version 0.9.