Download the PHP package avtonom/mathielen-import-engine without Composer

Mathielen Import Engine

Full-blown importer stack for importing almost any data into your application. Can be used for exports, too.

Introduction

This library implements some high-level functionality based on the great Ddeboer Data Import library. As the Data-Import library offers a great toolkit to implement a data import/export process in a quick and clean way, there is still a lot of work to do to have a full blown importer set up for your application. This library helps you with that.

If you are developing a Symfony2 project, you may want to use the comfortable configuration of the ImportEngineBundle.

Features

• A Storage Abstraction-layer that supports nice features like automatic delimiter-discovering or processing compressed files. Currently these storages are supported:
  • Structured Files
  • CSV, XML, JSON, Excel
  • File may be compressed
  • Doctrine2 queries
  • Service endpoints
• Storage Provisioning. Provide a list of possible storage-containers for your import. I.e. local files, remote files, uploaded files, database connections, service endpoints and more.
• A mapping sub-system, for building various mappings for your import: field-field, field-converter-field, field-converter-object and more.
• Automatic mapping into object tree's using the JMSSerializer
• Source (read) and Target (write) validation using Symfony Validation. Annotations can be used.
• Integrated Eventsystem using Symfony EventDispatcher
• Keeping almost every flexibility that is offered by the Ddeboer Data Import library.
• Well-tested code.

Installation

This library is available on Packagist. The recommended way to install it is through Composer:

Then include Composer’s autoloader:

If you want to make use of excel files, please also make sure to include phpoffice/phpexcel in your project:

Quickstart

Import an arbitrary file into your system

Using the *Provider facilities enables you to let the importer-system figure out what format the file has and what abstraction-classes should be used.

More examples

Have a look at: https://github.com/mathielen/import-engine/tree/master/tests/functional/Mathielen/ImportEngine

Usage

Terminology

• An Importer is the basic definition of the whole import-process. It says what may be imported and where to. It consists of:
  • (optional) A StorageProvider, that represents a "virtual file system" for selecting a SourceStorage
  • (optional) A SourceStorage that may be a file, a database table, an array, an object-tree, etc
  • A TargetStorage that may be a file, a database table, an array, an object-tree, etc
  • A Mapping, which may contain converters, field-mappings, etc
  • A Validation, that may contain validation-rules for data read from the SourceStorage and/or validation-rules for data that will be written to the TargetStorage.
  • An Logging or other interactions within the process.
• An Mapping.
• The ImportRunner is used to process the Import.
• Every run of an Import is represented by an ImportRun

StorageProvider

StorageProviders represents a "virtual file system" for selecting a SourceStorage that can be used as a source or target of the import.

FinderFileStorageProvider

Using the Symfony Finder Component as a collection of possible files that can be imported.

DoctrineQueryStorageProvider

You can use specific Doctrine Queries or only Entity-Classnames (the query will be SELECT * FROM then) as possible Source-Storages.

UploadFileStorageProvider

You can use a Provider to facilitate a File-Upload.

Automatic CSV Delimiter Discovery for FileStorageProviders

FileStorageProviders may use StorageFactories for constructing Storage objects. By default the FormatDiscoverLocalFileStorageFactory is used. This StorageFactory uses a MimeTypeDiscoverStrategy to determine the mime-type of the selected file and use it to create the correct storage-handler. You can change this behavior or extend it. There is a CsvAutoDelimiterTypeFactory that you can use to automaticly guess the correct delimiter of a CSV file.

This way any file that has the text/plain mime-type will be passed to the CsvAutoDelimiterFormatFactory to determine the delimiter.

Storage

A storage is a container of data. Storages provide a reader and writer implementation for itself.

Validation

You can get the source and target validation errors with:

Source data validation

Target data Validation

ClassValidatorFilter

You can use the ClassValidatorFilter to map the data to an object-tree and validate the objects (using annotations, or differently configurated validation rules). Therefore you must provide an ObjectFactory. There is a JmsSerializerObjectFactory you may want to use.

Importer

Import / Source Storage

You can either use a StorageProvider (see above) and set the selection-id or you can use a specific Storage-Handler directly:

Mapping

Also see orginal documentation here.

Converting fields

There are a some field-level build-in converters available:

• upperCase
• lowerCase
• @TODO

Custom fieldlevel-converting

You have to register more complex converters to the importer for selecting them in your import.

Custom rowlevel-converting

Like the fieldlevel converters, you have to register your converters first.

ImportRunner

For running a configured Import you need an ImportRunner. Internally the ImportRunner builds a workflow and runs it. You can change the way how the workflow is built by supplying a different WorkflowFactory.

ImportRun statistics

If you use the DefaultWorkflowFactory with your ImportRunner you get basic statistics from dryRun() and run() invocations.

Eventsystem

You can interact with the running import via the Symfony Eventdispatcher.

License

Import-Engine is released under the MIT license. See the LICENSE file for details.