Download the PHP package ttree/contentrepositoryimporter without Composer

ContentRepositoryImporter

This package contains generic utility to help importing data in the Neos Content Repository.

What's included ?

• A command controller (CLI) to launch your import presets
• Based on simple conventions
• DataProvider: used to prepare and cleanup data from the external source
• Importer: get the data from the DataProvider and push everything in the CR
• DataType: Simple object used to cleanup value and share code between DataProvider
• Split your import in multiple sub commands to avoid high memory usage
• No big magic, you can always take control by overriding the default configuration and methods

A basic DataProvider

Every data provider must extend the DataProvider abstract class or implement the interface . Check the source code of the abstract data provider, there are some useful things to discover.

It's important to update the property when you process data from the external source. During the processing, you can decide to skip some data (invalid data, missing values, ...) so we can not use the SQL count feature.

Try to do most of the data cleaning up in the data provider, so the data would arrive to the importer ready for insertion. Basically the array build by the provider should contains the data with the property name that match your node type property name. If you need to transport value that will not match the node properties, please prefix them with '_'.

There is some magic value, those values MUST be on the first level of the array:

• __identifier (optional) This UUID will be used in the imported node, you should use to have this feature, used by default
• __externalIdentifier (required) The external identifier of the data, this one is really important. The package keep track of imported data
• __label (required) The label of this record used by the importer mainly for logging (this value is not imported, but useful to follow the process) if you run twice the same import, the imported node will be updated and not created.

Tips: If the properties of your nodes are not at the first level of the array, you can override the method

Output of the provider

Your provider should output something like this:

Tips: If your provider does not return an array, you MUST registrer a TypeConverter to convert it to an array. The property mapper is used automatically by the Importer.

Content Dimensions support

If your data provider follow this convention, the importer can automatically create variants of your nodes:

The is a preset name, you must configuration the presets on your :

Share data between preset parts

You can split your import in multiple parts. Each parts is executed in a separate request. Sometimes it's useful to share data between parts (ex. in the first part you import the taxonomy, and in the second parts you map documents with the taxonomy). Those solve this use case, we integrate a feature called Vault. The Vault is simply a cache accessible in the importer and data provider by calling and . The current preset is the namespace, so you can use simple keys like name, ids, ...

The cache is flushed if you call .

Basic provider

A basic Importer

Every data importer must extend the AbstractImporter abstract class or implement the interface .

In the processRecord method you handle the processing of every record, such as creating Content Repository node for each incoming data record.

Do not forget to register the processed nodes with registerNodeProcessing. The method will handle feature like logging and tracking of imported node to decide if the local node need to be created or updated.

A basic preset

You can configure an import preset in your . A preset is split in multiple parts. If you use the , the current part will be executed by batch, by using a sub CLI request. This can solve memory or performance issue for big imports.

Start your import process

Tips: Do not forget to require this package from the package in which you do the importing, to ensure the correct loading order, so the settings would get overriden correctly.

From the CLI:

You can also filter the preset steps:

For testing purposes, or if you would like to override the value defined in your preset, you can also specify the number of records which should be imported at a time in an isolated sub-process:

Passing exceeding arguments to the DataProvider

The import process supports passing unnamed exceeding arguments to the DataProvider. This can be useful if you e.g. want to allow importing only a single record

Exceeding arguments will be available in the DataProvider through $this->getExceedingArguments(). You need to process this data yourself and apply it to your fetching logic.

Command based importers

Some data sources may consist of commands rather than data records. For example, a JSON file may contain create, update and delete instructions which reduce the guess-work on the importer's side, which records may be new, which should be updated and if the absence of a record means that the corresponding node should be deleted from the content repository.

For these cases you can extend the AbstractCommandBasedImporter. If your data records contain a mode field, the importer will try to call a corresponding command method within the same class.

Consider the following data source file as an example:

A corresponding ProductImporter might look like this:

CSV Data Provider

This package comes with a basic data provider for CSV files which will suffice for many scenarios. The class name for this data provider is Ttree\ContentRepositoryImporter\DataProvider\CsvDataProvider.

The following options can be passed to the data provider:

• csvFilePath: the full path and filename leading to the file to import
• csvDelimiter: the delimiter used in the CSV file (default: ,)
• csvEnclosure: the character which is used for enclosing the values (default: ")
• skipHeader: if the first line in the CSV file should be ignored (default: false)

Here is an example for a preset using the CSV Data Provider:

Acknowledgments

Development sponsored by ttree ltd - neos solution provider.

We try our best to craft this package with a lots of love, we are open to sponsoring, support request, ... just contact us.

License

Licensed under GPLv3+, see LICENSE