Download the PHP package code-rhapsodie/dataflow-bundle without Composer

Code Rhapsodie Dataflow Bundle

DataflowBundle is a bundle for Symfony 3.4+ providing an easy way to create import / export dataflow.

Dataflow uses a linear generic workflow in three parts:

• one reader
• any number of steps that can be synchronous or asynchronous
• one or more writers

The reader can read data from anywhere and return data row by row. Each step processes the current row data. The steps are executed in the order in which they are added. And, one or more writers save the row anywhere you want.

As the following schema shows, you can define more than one dataflow:

Features

• Define and configure a Dataflow
• Run the Job scheduled
• Run one Dataflow from the command line
• Define the schedule for a Dataflow from the command line
• Enable/Disable a scheduled Dataflow from the command line
• Display the list of scheduled Dataflow from the command line
• Display the result for the last Job for a Dataflow from the command line
• Work with multiple Doctrine DBAL connections

Installation

Security notice: Symfony 4.x is not supported before 4.1.12, see https://github.com/advisories/GHSA-pgwj-prpq-jpc2

Add the dependency

To install this bundle, run this command :

Suggest

You can use the generic readers, writers and steps from PortPHP.

For the writers, you must use the adapter CodeRhapsodie\DataflowBundle\DataflowType\Writer\PortWriterAdapter like this:

Register the bundle

Symfony 4 (new tree)

For Symfony 4, add CodeRhapsodie\DataflowBundle\CodeRhapsodieDataflowBundle::class => ['all' => true], in the config/bundles.php file.

Like this:

Symfony 3.4 (old tree)

For Symfony 3.4, add a new line in the app/AppKernel.php file.

Like this:

Update the database

This bundle uses Doctrine DBAL to store Dataflow schedule into the database table (cr_dataflow_scheduled) and jobs (cr_dataflow_job).

If you use Doctrine Migration Bundle or Phinx or Kaliop Migration Bundle or whatever, you can add a new migration with the generated SQL query from this command:

If you have already the tables, you can add a new migration with the generated update SQL query from this command:

Configuration

By default, the Doctrine DBAL connection used is default. You can configure the default connection. Add this configuration into your Symfony configuration:

By default, the logger service will be used to log all exceptions and custom messages. If you want to use another logger, like a specific Monolog handler, Add this configuration:

Messenger mode

Dataflow can delegate the execution of its jobs to the Symfony messenger component, if available. This allows jobs to be executed concurrently by workers instead of sequentially.

To enable messenger mode:

You also need to route Dataflow messages to the proper transport:

Define a dataflow type

This bundle uses a fixed and simple workflow structure in order to let you focus on the data processing logic part of your dataflow.

A dataflow type defines the different parts of your dataflow. A dataflow is made of:

• exactly one Reader
• any number of Steps
• one or more Writers

Dataflow types can be configured with options.

A dataflow type must implement CodeRhapsodie\DataflowBundle\DataflowType\DataflowTypeInterface.

To help with creating your dataflow types, an abstract class CodeRhapsodie\DataflowBundle\DataflowType\AbstractDataflowType is provided, allowing you to define your dataflow through a handy builder CodeRhapsodie\DataflowBundle\DataflowType\DataflowBuilder.

This is an example to define one class DataflowType:

Dataflow types must be tagged with coderhapsodie.dataflow.type.

If you're using Symfony auto-configuration for your services, this tag will be automatically added to all services implementing DataflowTypeInterface.

Otherwise, manually add the tag coderhapsodie.dataflow.type in your dataflow type service configuration:

yaml CodeRhapsodie\DataflowExemple\DataflowType\MyFirstDataflowType: tags:

• { name: coderhapsodie.dataflow.type } shell script $ bin/console debug:container --tag coderhapsodie.dataflow.type --show-private

Symfony Container Public and Private Services Tagged with "coderhapsodie.dataflow.type" Tag

Service ID Class name

CodeRhapsodie\DataflowExemple\DataflowType\MyFirstDataflowType CodeRhapsodie\DataflowExemple\DataflowType\MyFirstDataflowType

shell script $ SYMFONY_ENV=prod php bin/console code-rhapsodie:dataflow:run-pending shell script $ bin/console code-rhapsodie:dataflow:dump-schema --update --connection=default shell script

Run for dataflow DBAL connection

$ bin/console code-rhapsodie:dataflow:run-pending --connection=dataflow

Run for default DBAL connection

$ bin/console code-rhapsodie:dataflow:run-pending --connection=default

# Issues and feature requests

Please report issues and request features at https://github.com/code-rhapsodie/dataflow-bundle/issues.

Please note that only the last release of the 3.x and the 4.x versions of this bundle are actively supported.

# Contributing

Contributions are very welcome. Please see [CONTRIBUTING.md](CONTRIBUTING.md) for
details. Thanks to [everyone who has contributed](https://github.com/code-rhapsodie/dataflow-bundle/graphs/contributors)
already.

# License

This package is licensed under the [MIT license](LICENSE).