BoutDeCode Sylius ETL Plugin

Integrate a full ETL (Extract, Transform, Load) pipeline system into the Sylius admin panel.

## Overview This plugin adds a full-featured ETL workflow management system to the Sylius admin panel. It allows store administrators to define reusable **Workflows** (named chains of ordered ETL Steps), and execute them as **Pipelines** — either manually, via file upload, or on a schedule. The plugin provides the ETL infrastructure (workflow management, pipeline execution, history tracking). Step implementations are defined in your application. ## Features - **Workflow management** — create and configure reusable chains of ETL steps from the admin panel - **Pipeline execution** — run workflows as pipelines with JSON input, file upload, or scheduled execution - **Execution history** — detailed per-step and per-pipeline run history with status tracking - **Step registry** — register your own step services tagged with `etl.step` - **State machine** — pipeline lifecycle management with reset/execute transitions - **Planned tasks** — define recurring ETL jobs by binding a workflow to a cron expression; a new pipeline is created and dispatched automatically on each occurrence - **Dedicated logging** — separate `pipeline` log channel for ETL activity - **Notifications** — notify on pipeline success and/or failure per workflow, through pluggable providers (built-in email support) - **Pipeline purge** — opt-in scheduled cleanup of old completed/failed pipelines and their history, to keep storage bounded - **Sylius admin integration** — ETL section added to the admin sidebar with grid views for Workflows and Pipelines ### Screenshots **Workflow list**  **Workflow editor** — configure ordered steps with their configuration  **Pipeline list**  **Pipeline detail** — inspect pipeline and configuration  **Execute a pipeline** — provide a file upload, JSON input or a scheduled date; the workflow configuration can also be overridden at runtime per step  **Pipeline run history** — per-step status and execution timeline  ## Requirements - PHP `^8.1` - Sylius `^1.14` - Symfony `^6.4` - Symfony Messenger with an `async` transport ## Installation ### 1. Require the plugin via Composer ### 2. Enable the plugin Add the plugin to your `config/bundles.php`: ### 3. Import the plugin configuration **PHP** — create or update `config/packages/bout_de_code_sylius_etl.php`: **YAML** — create or update `config/packages/bout_de_code_sylius_etl.yaml`: ### 4. Import the admin routes **PHP** — add to `config/routes.php` (or `config/routes/sylius_admin.php`): **YAML** — add to `config/routes.yaml` (or `config/routes/sylius_admin.yaml`): ### 5. Configure Symfony Messenger Ensure an `async` transport is configured in `config/packages/messenger.yaml`: ### 6. Import the admin assets Add a dedicated entry in your `webpack.config.js`: Then include the compiled entry in your admin layout (e.g. `templates/bundles/SyliusAdminBundle/layout.html.twig`): Then install and build the assets: ### 7. Run database migrations ### 8. Start the Messenger worker Pipeline execution and planned task scheduling are processed asynchronously via Symfony Messenger. Start the worker with both the `async` and `scheduler_etl` transports: In production, use a process manager (Supervisor, `systemd`, etc.) to keep the worker running continuously. ## Usage ### Workflows Navigate to **Admin > ETL > Workflows** to create a new Workflow. A workflow defines an ordered chain of steps, each with a step type code and a JSON configuration. Step type codes correspond to services tagged `etl.step` registered in your application. See the [etl-core documentation](https://github.com/boutdecode/etl-core) for instructions on how to create and register custom steps. #### Notifications Each workflow can be configured to notify on success and/or on failure, and to restrict which providers are used (leave empty to notify through every registered provider). The built-in `email` provider is configured globally in your application, not per workflow: See the [etl-core documentation](https://github.com/boutdecode/etl-core#notifications) for how to implement a custom notification provider (e.g. Slack). ### Pipelines Navigate to **Admin > ETL > Pipelines** to create and execute pipelines from existing workflows. When creating a pipeline you can provide: - A **JSON input** payload - A **file upload** (e.g. a CSV file) - A **JSON configuration override** to customize step parameters at runtime - A **scheduled date/time** for deferred execution - A **cron expression** for recurring scheduled execution ### Purging old Pipelines The bundle can automatically delete old pipeline runs so that storage doesn't grow unbounded. This is handled by a scheduled job (disabled by default) that removes pipelines in a terminal state (`completed` or `failed`) whose finish date is older than a configurable retention window, together with their `PipelineHistory` and `StepHistory` records. `WorkflowStatistic` and `WorkflowExecutionStatistic` records are left untouched, so aggregated reporting survives pipeline purges. See the [etl-core documentation](https://github.com/boutdecode/etl-core#purging-old-pipelines) for details. ### Planned tasks Navigate to **Admin > ETL > Planned tasks** to create recurring ETL jobs. A planned task binds a **Workflow** to a **cron expression** (e.g. `0 2 * * *` for every day at 2 AM). Every minute the built-in Symfony Scheduler fires, picks up all enabled planned tasks, and dispatches an execution command for each one. The runner then: 1. Creates a new `Pipeline` from the task's workflow, applying any per-task configuration and input overrides. 2. Computes the next scheduled run date from the cron expression. 3. Persists the pipeline and links it back to the planned task. **Available fields when creating a planned task:** | Field | Description | |---|---| | Name | Human-readable label | | Workflow | The workflow to execute | | Schedule | Cron expression (e.g. `*/5 * * * *`) | | Enabled | Toggle to activate/deactivate without deleting | | Configuration | JSON override merged on top of the workflow's step configuration | | Input | Inline JSON payload or a file upload used as pipeline input | ### Logging Pipeline execution is logged to a dedicated file: ## Testing ### PHPUnit ### Behat (non-JS scenarios) ### Behat (JS scenarios) 1. [Install Symfony CLI](https://symfony.com/download). 2. Start Headless Chrome: 3. Start the test application server: 4. Run Behat: ### Static Analysis ### Coding Standards ## License This plugin is released under the [MIT License](LICENSE).