Download the PHP package aymdev/fregata without Composer

Fregata - PHP database migrator

Fregata is a data migration framework. You can use it to migrate any kind of data, but it has features to help you migrate between different DBMS or database structures.

Documentation:

1. Introduction
2. Setup
   1. Installation
   2. Configuration
      1. Kernel and service container
      2. YAML configuration
3. Components
   1. Migration Registry
   2. Migration
      1. Options
      2. Parent migration
   3. Task
   4. Migrator
      1. Puller
      2. Pusher
      3. Executor
4. Tools
   1. Migration Context
5. Features
   1. Dependent migrators
   2. Batch pulling
   3. Foreign Key migrations
6. CLI usage
   1. List migrations
   2. Get details of a migration
   3. Execute a migration
7. Contributing

Introduction

Fregata is a data migration framework. It can probably be compared to an ETL (Extract Transform - Load) tool.

You can use it to migrate data from files, databases, or anything you want, it is completely agnostic on this part (some of its test migrate data between PHP arrays). But note that it was initially targeting databases, providing a way to migrate data between different DBMS, even with a different structure. Some included features are specifically built for databases.

Why creating a framework for data migration ?

While database migrations might not be your everyday task, I encountered it multiple times on different projects. That's why I created Fregata to have a migration workflow I could reuse.

What are the use cases ?

Here are some example use cases (from experience):

• when you want to change from a DBMS to another
• when you want to sync your staging database with the production one (useful for CMS-based projects)

Setup

Installation

Install with Composer:

Configuration

Fregata expects you to have a config and a cache directory at your project root by default.

Kernel and service container

If you need to use a different directory structure than the default one, you can extend the Fregata\Configuration\AbstractFregataKernel class. Then you will have to implement methods to specify your configuration and cache directory.

Important: your kernel full qualified class name must be App\FregataKernel.

The kernel holds a service container, built from Symfony's DependencyInjection component. This means you can define your own services as you would do it in a Symfony application, in a services.yaml file in your configuration directory.

Here's a recommended minimal services.yaml to start your project:

YAML configuration

To configure Fregata itself, you will need a fregata.yaml file in your configuration directory.

Example configuration file:

Components

Migration Registry

The migration registry contains every defined migrations. You shouldn't have to interact with it.

Migration

A migration project holds the steps of a migration. For example, data migration from your production database to staging one. Each migration is created and saved into the registry based on your configuration. You don't need to instantiate migration objects by yourself.

Migrations contain tasks and migrators. When a migration is run, components are executed in the following order:

• before tasks
• migrators
• after tasks

Options

You may need to set specific configuration to your migration project, which could be used by tasks or migrators. With the options key you can define your migration specific configuration, they will be accessible to the components from the migration context.

Parent migration

When having multiple migrations for different environments, you probably want to avoid duplicating your whole configuration. You can extend a migration with the parent key. The "child" migration will inherit the parent's options, tasks and migrators. You can still add more tasks and migrators, and overwrite options.

Task

A task can be executed before or after migrators. They can be useful to bootstrap your migration (before tasks) or to clean temporary data at the end (after tasks):

Migrator

The migrators are the main components of the framework. A single migrator holds 3 components:

• a puller
• a pusher
• an executor

It must return its components from getter methods by implementing Fregata\Migration\Migrator\MigratorInterface. A migrator represents the migration of a data from a source to a target. For example, migrating data from a MySQL table to a PostgreSQL one.

Puller

A puller is a migrator component responsible for pulling data from a source. It returns data and optionally the number of items to migrate:

Pusher

A pusher gets item fetched by the puller 1 by 1 and has to push the data to a target:

Here $data is a single item from the example puller returned value. The push() method is called multiple times. The separation of pullers and pushers allow you to migrate between different sources: pull from a file and push to a database, etc.

Executor

The executor is the component which plugs a puller with a pusher. A default one is provided and should work for most cases: Fregata\Migration\Migrator\Component\Executor. Extend the default executor if you need a specific behaviour.

Tools

Migration Context

You can get some informations about the current migration by injecting the Fregata\Migration\MigrationContext service in a task or migration.

It provides:

• current migration object
• current migration name
• migration options
• parent migration name if applicable

Features

Dependent migrators

If your migrators need to be executed in a specific order you can define dependencies, and they will be sorted automatically:

Here, DependencyMigrator will be executed before DependentMigrator.

Batch pulling

When a puller works with very large datasets you might want to pull the data by chunks:

Foreign Key migrations

One of the most complex parts of a database migration is about foreign keys. There are multiple steps to follow to perform a valid foreign key migration. This is done using Doctrine DBAL.

You must add 2 tasks to your migration:

• before task: Fregata\Adapter\Doctrine\DBAL\ForeignKey\Task\ForeignKeyBeforeTask
• after task: Fregata\Adapter\Doctrine\DBAL\ForeignKey\Task\ForeignKeyAfterTask

The before task will create temporary columns in your target database to keep the original referenced and referencing columns. It may also change referencing columns to allow NULL (only if you specify it). The after task will set the real values in your original referencing columns and then drop the temporary columns.

Then the migrators must provide the database connection and the list of foreign keys:

The migrators are responsible for the data migration, this means you need to fill the temporary columns with original primary/foreign key from the source database. To get the name of a temporary column, require the CopyColumnHelper service in your pusher:

This example show the local (or referencing) side but this need to be done for the foreign (or referenced) side too, using CopyColumnHelper::foreignColumn().

CLI usage

Fregata provides a simple program to run the migrations, you can launch it with:

List migrations

To list the migrations of your installation, run the migration:list command:

Get details of a migration

To get information about a single migration, run the migration:show command:

Execute a migration

And the most important one to run a migration: migration:execute.

Contributing

A Docker setup is available, providing a MySQL 5.7 service.

If you want to test the implementation of the framework (using a Composer path repository), install it in a _implementation directory at the root of the project, it is ignored by Git by default and will ensure you are using your implementation autoloader.