Download the PHP package sandstorm/e2etesttools without Composer

End-To-End Test Tools

... for Neos and Flow Projects.

for SYMFONY projects, see README.Symfony.md.

We use Playwright as browser orchestrator for tests involving a real browser. We use Behat as the test framework for writing all kinds of BDD tests.

• a way to test Fusion code with Behat

• Utilities for integrating the Playwright browser orchestrator and the Behat test framework

• End-To-End Test Tools
• Installation and Setup Instructions
  • Setting up Playwright
  • Creating a FeatureContext
  • Loading CSS and JavaScript for the Styleguide
  • Pipeline Setup
• Running Behat Tests
  • Style Guide
• Writing Behat Tests
  • Fusion Component Testcases
  • Fusion Integration Testcases
  • Full-Page Snapshot Testcases
• Architecture

Installation and Setup Instructions

• either copy .mise.toml and update it, then run mise run e2e:setup

• or do the following:

• you can delete behat.yml and only keep behat.yml.dist

• in behat.yml.dist, remove the Behat\MinkExtension part completely.

  Mink is generic a "browser controller API" which in our experience is a bit brittle to use and adds unnecessary complexity. We recommend to instead use Playwright directly.

• You should configure the Flow/Neos Configuration/Testing/Behat/Settings.yaml and copy the production Settings.yaml there; to ensure that Behat is accessing the same Database like the production application.

• You should create a Configuration/Development/Docker/Behat/Settings.yaml with the following contents:

• You should create a Configuration/Production/Kubernetes/Behat/Settings.yaml with the following contents:

Setting up Playwright

We suggest copying Resources/Private/e2e-testrunner-template of this package to the root of the Git Repository and name the folder e2e-testrunner (in our projects, usually one level ABOVE the Neos Root Directory). Also, make sure you have installed playwright on your device. npx playwright install

Creating a FeatureContext

The FeatureContext is the PHP class containing the step definitions for the Behat scenarios. We provide base traits you should use for various functionality. We provide a working skeleton for you to use under Tests/Behavior/Bootstrap/FeatureContext.php.default Copy this file as your FeatureContext.php to your project Tests/Behavior/Bootstrap/FeatureContext.php cp -R ./Packages/Application/Sandstorm.E2ETestTools/Tests/Behavior/Bootstrap/FeatureContext.php.default ./DistributionPackages/<Your.PackageName>/Tests/Behavior/Bootstrap/FeatureContext.php Inside the file, check the paths to the provided Sandstorm traits and update if necessary.

Loading CSS and JavaScript for the Styleguide

In your Fusion code, add the JavaScript and CSS of your page to the Sandstorm.E2ETestTools:StyleguideStylesheets and Sandstorm.E2ETestTools:StyleguideJavascripts prototypes, e.g. in the following way:

Additionally, the base URL needs to be configured correctly. This package sets it to "/" in the Testing/Behat context which will work in most cases out of the box.

Pipeline Setup

We provide a skeleton to run e2e tests in your gitlab pipeline. Add the provided lines from our .gitlab-ci.yml to yours and adjust accordingly.

Every related service (like redis, database, ...) needs to be started using a servives entry. Ensure the Docker image version of the service matches the development and production image from docker-compose.yml.

The environment variables of the job are passed on to all services - so all connected services and the main job share the same environment variables. Thus, you need to add the environment variables for BOTH the SUT (which is the main job) and all related services to the variables section of the test job.

Running Behat Tests

This is MANDATORY to read for everybody. We suggest that this section is COPIED to the readme of your project.

First, you need to start the Playwright Server on your development machine. For that, go to e2e-testrunner in your Git Repo, and do:

Second, ensure the docker containers are running; usually by docker-compose build && docker-compose up -d. Then, enter the neos container: docker-compose exec neos /bin/bash and run the following commands inside the container:

Behat also supports running single tests or single files - they need to be specified after the config file, e.g.

In case of exceptions, it might be helpful to run the tests with --stop-on-failure, which stops the test cases at the first error. Then, you can inspect the testing database and manually reproduce the bug.

Additionally, -vvv is a helpful CLI flag (extra-verbose) - this displays the full exception stack trace in case of errors.

For hints how to write Behat tests, we suggest to read Sandstorm.E2ETestTools README.

Style Guide

If you use the Style Guide feature (Then I store the Fusion output in the styleguide as "Button_Component_Basic"), then your tests need to be annotated with @playwright and the playwright dev server needs to be running.

You can then access the style guide using 127.0.0.1:8080/styleguide/. The style guide contains BOTH HTML snapshots; and rendered images of the HTML.

Writing Behat Tests

Here, we try to give examples for common Behat scenarios; such that you can easily get started.

Fixture Setup

The Sandstorm.E2ETestTools Package provides inline, delegated and hybrid fixture setups. We recommend using a hybrid approach.

Delegated / Hybrid

In your Neos Backend, select a node you want to test, go to the meta tab and click "export node". This will download a yaml file containing all the selected node's parents and all descendants of the nearest document parent (in case of dependencies as such references). Afterwards, move the downloaded yaml file into test directory and use them in your .feature file like such:

Also, you can override node properties inline:

Inline

Fusion Component Testcases

You can use a test case like the following for testing components - analogous to what you usually do with Monocle.

Some hints:

• We need to set up a minimal node tree, as otherwise we cannot render links.

Fusion Integration Testcases

It is especially valuable to not just test the Fusion component (which is more or less like a pure function), but instead test that a given Node renders in a certain way - so that the wiring between Node and Fusion component is set up correctly.

A test case can look like the following one:

Full-Page Snapshot Testcases

This tests a complete page rendering, and not just single components. It is meant mostly for visual checking; and most likely you'll work less with specific assertions.

In this case, the rendering depends on many more nodes - so setting up the behat fixture with all the relevant nodes can be a bit tedious. Luckily, there are helpers in this package to help with the process. We suggest writing a CommandController like the following:

Now, when you run ./flow stepGenerator:homepage, you'll get a table like the following:

This is ready to be pasted into a test case like the following:

This enables to generate responsive, reproducible screenshots of the different pages, and being able to re-generate this when the dummy data changes.

persistent resources in BDD tests

In case, your node fixtures point to some assets from the Neos.Media module, you can generate fixtures for them as well. You need to pass the second parameter ($fixtureBasePath) when creating a NodeTable.

You probably want to store you asset fixtures near your feature files.

TODO explain how to set fixture base path

Let's say you have three images in your node data fixtures (node property of type ImageInterface). Your output could look like:

Note, that the Path column values are printed and read relative to the Flow package directory. That should keep your tests more or less environment independent. Usually, the files are stored inside a DistributionPackages/* package which is symlinked into the Flow package directory (and thus is readable from your Test and writable from your Command Controller). Also, those files should be added to git, since they are part of your test cases.

dynamic modification of SUT URL via step

By default, the SUT URL is configured statically via environment variable. In some cases, that is not sufficient.

Use cases:

custom content dimension resolving based on host info

Let's say, your Neos project has a custom content dimension value resolver, f.e. by host name or subdomain. The SUT base URL is configured statically via environment variable. But in the mentioned special case, you need dynamic base URLs that are modified via your own custom steps.

multi-site setup

When your Neos application has multiple sites, the host name also needs to be defined via custom step.

The PlaywrightConnector has an API for that purpose:

public API: PlaywrightTrait#setSystemUnderTestUrlModifier(\Closure $urlModifier): void delegates to internal: PlaywrightConnector#setSystemUnderTestUrlModifier(\Closure $urlModifier): void

Note, that the modifier is reset after each scenario.

You need to call that setter from your custom step, that could look like:

and behat call:

Usage for Site Packages that use Sandstorm.NeosAcl

add this to your Policy.yaml in the Testing/Behat context:

pause for debugging

If you want to use the pause functionality of playwright, please start the test with PAUSE_FOR_DEBUGGING=true to prevent curl timeouts when communicating with the e2e-testrunner.

Architecture

We suggest to skim this part roughly to get an overview of the general architecture. As long as you do not do in-depth modifications, you do not need to read it in detail.

The architecture for running behavioral tests is as follows:

1) We add the Behat test runner to the Development or Production App Docker Container (SUT - System under Test), so that the Behat test runner can access any code from the application, and has the exact same environment, database, and library versions like the production application.

2) The E2E Testrunner wraps Playwright (which is a browser orchestrator) and exposes a HTTP API. It is running as associated service. Behat communicates to the test runner via HTTP (1).

3) Then, the testrunner calls the unmodified application via HTTP (2).

4) The application then calls other services like Redis and the database - just as usual.

There is one catch with big implications, though: The E2E tests need full control over the database to work reliably. As we do not want to clear our development database each time we run our tests, we need to use two databases: one for Testing, and the other one for Development.

Additionally, the E2E tests need to reach the system wired to the testing environment through HTTP. This means we need two web server ports as well: One for development, and one for the testing context.

This setup is somewhat complicated; so the following image helps to illustrate how the different contexts interact during development time and during production/CI: