Download the PHP package silverstripe/behat-extension without Composer

Silverstripe Integration for Behat

Overview

Behat is a testing framework for behaviour-driven development. Because it primarily interacts with your website through a browser, you don't need any specific integration tools to get it going with a basic Silverstripe website, simply follow the standard Behat usage instructions.

This extension comes in handy if you want to go beyond interacting with an existing website and database, for example make changes to your database content which would need to be rolled back to a "clean slate" later on.

It provides the following helpers:

• Provide access to Silverstripe classes in your Behat contexts
• Set up a temporary database automatically
• Reset the database content on every scenario
• Prebuilt Contexts for SilverStripe's login forms and other common tasks
• Creating of member fixtures with predefined permissions
• YML fixture definitions inside your Behat scenarios
• Waiting for jQuery Ajax responses (rather than fixed wait timers)
• Captures JavaScript errors and logs them through Selenium
• Saves screenshots to filesystem whenever an assertion error is detected

In order to achieve this, the extension makes one basic assumption: Your Behat tests are run from the same application as the tested Silverstripe codebase, on a locally hosted website from the same codebase. This is important because we need access to the underlying SilverStripe PHP classes. You can of course use a remote browser to do the actual testing.

Note: The extension has only been tested with the selenium2 Mink driver.

Installation

In a Silverstripe CMS project (see getting started docs) add the Silverstripe Behat extension via Composer.

Download the standalone Google Chrome WebDriver

Unless you have SS_BASE_URL set up, you also need to specify the URL for your webroot. Either add it to the existing behat.yml configuration file in your project root, or set is as an environment variable in your terminal session:

Usage

Starting ChromeDriver

You can run the server locally in a separate Terminal session:

Running the Tests

Now you can run the tests (for example for the framework module):

Or even run a single scenario by it's name (supports regular expressions):

This will start a Chrome browser by default. Other browsers and profiles can be configured in behat.yml.

Running with stand-alone command (requires Bash)

If running with silverstripe/serve and chromedriver, you can also use the following command which will automatically start and stop these services for individual tests.

This automates:

• starting server
• starting chromedriver
• running behat
• shutting down chromedriver
• shutting down server

Make sure you set SS_BASE_URL to http://localhost:8080 in .env

Tutorials

• Tutorial: Testing Form Submissions
• Tutorial: Webservice Mocking with Phockito and TestSession
• Tutorial: Setting up Behat on CircleCI

Configuration

The Silverstripe installer already comes with a YML configuration which is ready to run tests on the standalone ChromeDriver server, located in the project root as behat.yml.

You should ensure that you have configured SS_BASE_URL in your .env file.

Generic Mink configuration settings are placed in SilverStripe\BehatExtension\MinkExtension, which is a subclass of Behat\MinkExtension\Extension.

Overview of settings (all in the extensions.SilverStripe\BehatExtension\Extension path):

• ajax_steps: Because Silverstripe uses AJAX requests quite extensively, we had to invent a way to deal with them more efficiently and less verbose than just Optional ajax_steps is used to match steps defined there so they can be "caught" by special AJAX handlers that tweak the delays. You can either use a pipe delimited string or a list of substrings that match step definition.
• ajax_timeout: Milliseconds after which an Ajax request is regarded as timed out, and the script continues with its assertions to avoid a deadlock (Default: 5000).
• screenshot_path: Absolute path used to store screenshot of a last known state of a failed step. Screenshot names within that directory consist of feature file filename and line number that failed.

Example: behat.yml

Module Initialisation

You're all set to start writing features now! Simply create *.feature files anywhere in your codebase, and run them as shown above. We recommend the folder structure of tests/behat/features, since its consistent with the common location of SilverStripe's PHPUnit tests.

Behat tests rely on a FeatureContext class which contains step definitions, and can be composed of other subcontexts, e.g. for SilverStripe-specific CMS steps (details on behat.org). Since step definitions are quite domain specific, its likely that you'll need your own context. The Silverstripe Behat extension provides an initializer script which generates a template in the recommended folder structure:

Note: namespace is mandatory

You'll now have a class located in mymodule/tests/behat/src/FeatureContext.php, which will have a psr-4 class mapping added to composer.json by default. Also a folder for your features with mymodule/tests/behat/features will be created. A mymodule/behat.yml is built, with a default suite named after the module.

Available Step Definitions

The extension comes with several BehatContext subclasses come with some extra step defintions. Some of them are just helpful in general website testing, other's are specific to SilverStripe. To find out all available steps (and the files they are defined in), run the following:

Note: There are more specific step definitions in the Silverstripe framework module for interacting with the CMS interfaces (see framework/tests/behat/features/bootstrap). In addition to the dynamic list, a cheatsheet of available steps can be found at the end of this guide.

Fixtures

Since each test run creates a new database, you can't rely on existing state unless you explicitly define it.

Database Defaults

The easiest way to get default data is through DataObject->requireDefaultRecords(). Many modules already have this method defined, e.g. the blog module automatically creates a default BlogHolder entry in the page tree. Sometimes these defaults can be counterproductive though, so you need to "opt-in" to them, via the @database-defaults tag placed at the top of your feature definition. The defaults are reset after each scenario automatically.

Inline Definition

If you need more flexibility and transparency about which records are being created, use the inline definition syntax. The following example shows some syntax variations:

• Fixtures are created where you defined them. If you want the fixtures to be created before every scenario, define them in Background. If you want them to be created only when a particular scenario runs, define them there.
• Fixtures are cleared between scenarios.
• The basic syntax works for all DataObject subclasses, but some specific notations like "is not published" requires extensions like Hierarchy to be applied to the class
• Record types, identifiers, property names and property values need to be quoted
• Record types (class names) can use more natural notation ("registration page" instead of "Registration Page")
• Record types support the $singular_name notation which is also used to reference the types throughout the CMS. Record property names support the $field_labels notation in the same fashion.
• Property values may also use a => symbol to indicate relationships between records. The notation is =><classname>.<identifier>. For has_many or many_many relationships, multiple relationships can be separated by a comma.

Writing Behat Tests

Directory Structure

As a convention, Silverstripe Behat tests live in a tests/behat subfolder of your module. You can create it with the following commands:

FeatureContext

The generic Behat usage instructions apply here as well. The only major difference is the base class from which to extend your own FeatureContext: It should be SilverStripeContext rather than BehatContext.

Example: mymodule/tests/behat/src/FeatureContext.php

Screen Size

In some Selenium drivers you can define the desired browser window size through a capabilities definition. By default, Selenium doesn't support this though, so we've added a workaround through an environment variable:

Inspecting PHP sessions

Behat is executed from CLI, which in turn triggers web requests in a browser. This browser session is associated PHP session information such as the logged-in user. After every request, the session information is persisted on disk as part of the TestSessionEnvironment, in order to share it with Behat CLI.

Example: Retrieve the currently logged-in member

FAQ

FeatureContext not found

This is most likely a problem with Composer's autoloading generator. Check that you have "SilverStripe" mentioned in the vendor/composer/autoload_classmap.php file, and call composer dump-autoload if not.

How do I wait for asynchronous actions in my steps?

Sometimes you want to wait for an AJAX request or CSS animation to complete before calling the next step/assertion. Mink provides a wait() method for this purpose - just let the execution wait until a JavaScript expression satisfies your criteria. It's pretty common to make this expression a CSS selector. The Behat tests come with built-in support to wait for any pending jQuery.ajax() requests, check BasicContext->handleAjaxBeforeStep() and the ajax_steps configuration option.

Why does the module need to know about the framework path on the filesystem?

Sometimes Silverstripe needs to know the URL of your site. When you're visiting your site in a web browser this is easy to work out, but if you're executing scripts on the command-line, it has no way of knowing.

How does the module interact with the SS database?

The module creates temporary database on init and is switching to the alternative database session before every scenario by using /dev/tests/setdb TestRunner endpoint.

It also populates this temporary database with the default records if necessary.

It is possible to include your own fixtures, it is explained further.

Why do tests pass in a fresh installation, but fail in my own project?

Because we're testing the interface directly, any changes to the viewed elements have the potential to disrupt testing. By building a test database from scratch, we're trying to minimize this impact. Some examples where things can go wrong nevertheless:

• Thirdparty Silverstripe modules which install default data
• Changes to the default interface language
• Configurations which remove admin areas or specific fields

Currently there's no way to exclude offending modules from a test run. You either have to adjust the tests to work around these changes, or run tests on a "sandbox" projects without these modules.

How do I debug when something goes wrong?

First, read the console output. Behat will tell you which steps have failed.

Silverstripe Behaviour Testing Framework also notifies you about some events. It tries to catch some JavaScript errors and AJAX errors as well although it is limited to errors that occur after the page is loaded.

Screenshot will be taken by the module every time the step is marked as failed. Refer to configuration section above to know how to set up the screenshot path.

If you are unable to debug using the information collected with the above methods, it is possible to delay the step execution by adding the following step:

And I put a breakpoint

This will stop the execution of the tests until you press the return key in the terminal. This is very useful when you want to look at the error or developer console inside the browser or if you want to interact with the session page manually.

Can I set breakpoints through XDebug?

If you have XDebug set up, breakpoints are your friend. The problem is that you can only connect the debugger to the PHP execution in the CLI, or in the browser, not both at the same time.

First of all, ensure that xdebug.remote_autostart is set to Off, otherwise you'll always have an active debugging session in CLI, never in the browser.

Then you can choose to enable XDebug for the current CLI run:

Or you can use the TESTSESSION_PARAMS environment variable to pass additional parameters to dev/testsession/start, and debug in the browser instead.

The macgdbp IDE key needs to match your xdebug.idekey php.ini setting.

How do I set up continuous integration through Travis?

Check out the travis.yml in silverstripe/framework for a good example on how to set up Behat tests through travis-ci.org.

Cheatsheet

This is a manually categorized list of available commands when both the cms and framework modules are installed. It's based on the vendor/bin/behat -di @cms output.

Basics

Navigation

Forms

Interactions

Login

CMS UI

Fixtures

Environment

Email

Transformations

Behat transformations have the ability to change step arguments based on their original value, for example to cast any argument matching the \d regex into an actual PHP integer.

• /^(?:(the|a)) time of (?<val>.*)$/: Transforms relative time statements compatible with strtotime(). Example: "the time of 1 hour ago" might return "22:00:00" if its currently "23:00:00".
• /^(?:(the|a)) date of (?<val>.*)$/: Transforms relative date statements compatible with strtotime(). Example: "the date of 2 days ago" might return "2013-10-10" if its currently the 12th of October 2013.
• /^(?:(the|a)) datetime of (?<val>.*)$/: Transforms relative date and time statements compatible with strtotime(). Example: "the datetime of 2 days ago" might return "2013-10-10 23:00:00" if its currently the 12th of October 2013.

Useful resources

• Silverstripe CMS architecture
• Silverstripe Framework Test Module
• Silverstripe Unit and Integration Testing