Download the PHP package pollora/datamorph without Composer

Datamorph

Datamorph is a Laravel package that allows you to create and run Flow PHP ETL (Extract, Transform, Load) pipelines in a structured and extensible way. This documentation will guide you through the installation, configuration, and usage of the package.

Table of Contents

• Installation
• Concepts
• Configuration
• Creating an ETL Pipeline
• Running an ETL Pipeline
• Hooks
• Concrete Examples

Installation

Install the package via Composer:

Concepts

Datamorph is built around three main components:

1. Extractors: Retrieve data from various sources (databases, APIs, files, etc.)
2. Transformers: Transform the retrieved data according to your needs
3. Loaders: Load the transformed data to their final destination

These three components are orchestrated in a Pipeline that also manages Hooks that allow you to intervene at different stages of the process.

Configuration

Publish the configuration file:

This will create a config/datamorph.php file where you can configure your ETL pipelines:

Creating an ETL Pipeline

Automatic File Generation

Datamorph includes an Artisan command that automatically generates the necessary files for a new pipeline:

This command will create the following files in the app/ETL/Product/ directory:

• ProductExtractor.php - For data extraction
• ProductTransformer.php - For data transformation
• ProductLoader.php - For loading transformed data

Structure of Generated Files

Extractor

Transformer

Loader

Running an ETL Pipeline

Once your components are implemented and your pipeline is configured, you can run it with the Artisan command:

This command:

1. Checks that the pipeline exists in the configuration
2. Checks that the Extractor, Transformer, and Loader classes exist
3. Instantiates these classes and creates a Pipeline
4. Runs the Pipeline with the configured hooks

Running in Code

You can also run a pipeline programmatically:

Hooks

Hooks are a powerful mechanism in Datamorph that allows you to intervene at different stages of an ETL pipeline. There are three ways to implement hooks in Datamorph, each with its own use cases.

Configuration-based Hooks

The first approach is to define hooks in the config/datamorph.php configuration file. This method is ideal for recurring hooks that need to be applied to every pipeline execution.

Configuration

Hook Implementation

Each hook must implement the HookInterface:

Dynamic Hooks

The second approach uses the $context->pipeline->after() methods to register hooks dynamically during pipeline execution. This method is particularly useful for conditional behaviors or hooks that depend on the current state of the pipeline.

Usage

Supported Hook Types

You can pass different types of hooks to the after() method:

1. A Closure (anonymous function) - Will be automatically wrapped in a DynamicHook
2. An instance of a class implementing HookInterface - Will be used directly
3. A class name - The class will be resolved via Laravel's IoC container

Automatic Operation Detection

If you don't explicitly specify the operation, Datamorph will detect it automatically based on the calling context:

• In an Extractor, the operation will be extract
• In a Transformer, the operation will be transform
• In a Loader, the operation will be load

Hooks via before/after Methods

The third approach is to directly implement the before() and after() methods in your Extractor, Transformer, and Loader classes. This method is the simplest and most direct for standard behaviors.

Implementation

The before() and after() methods are automatically called by the pipeline at the appropriate times, without any additional configuration.

Execution Order and Combining Approaches

All three approaches can be combined in the same pipeline. The execution order is as follows:

1. Hooks configured in config/datamorph.php
2. The before() method of the relevant ETL component
3. Main operation (extraction, transformation, loading)
4. The after() method of the relevant ETL component
5. Dynamic hooks registered via $pipeline->before() and $pipeline->after()

This combination offers great flexibility and can address a variety of use cases.

Usage Examples

Example 1: Configuration for Validation Before Extraction

Example 2: Dynamic Hooks for Conditional Logging

Example 3: Before/After Methods for Connection Management

Example 4: Combining Approaches for a Complete Pipeline

This combination of approaches provides you with a flexible and powerful hook system capable of addressing a variety of needs in your ETL pipelines.