Download the PHP package itx/importer without Composer
On this page you can find all versions of the php package itx/importer. It is possible to download/install these versions without Composer. Possible dependencies are resolved automatically.
Package importer
Short Description A TYPO3 extension which contains a failure safe framework for building structured import processes.
License GPL-2.0-or-later
Homepage https://www.itx.de/
Informations about the package importer
Importer Extension
This TYPO3 extension provides a failure tolerant structured import system for TYPO3 CMS. It provides a framework to create import jobs which, can be run in parallel. For every import there is a job queue that will be created by a producer class and consumed by a consumer class. The extension also provides a backend module to view and manage the imports, its jobs, statistics and potential failures.
Usage
Running the queue worker
To run the queue worker for the importer extension, run the following command:
vendor/bin/typo3cms importer:queue-worker
This will run a single queue worker process. You can run multiple queue workers in parallel to speed up the import.
To have them run automatically in multiple instances, you can use the following command:
vendor/bin/typo3cms importer:queue-worker-manager <optional worker count>
Importing data
To create a new import, you need to create 3 files:
-
Producer class
The producer class should inherit from the Itx\Importer\Command\Producer\AbstractJobProducer class. Inside the producer class you make the api calls to get information about the data you want to import. Finally, you will need to use the generateJobs method to create the jobs that will be processed by the queue worker.
-
Payload class
Use a custom payload class to pass typesafe data to the consumer class. This class doesn't getters and setters.
-
Consumer class
The consumer class will be called by the queue worker and needs to implement the ConsumerInterface. It will receive the payload class as a parameter. Inside the consumer class you can use the data from the payload class to import the data into your database. You don't need to register the consumer, it will be automatically detected.
When all jobs were processed the importer will call the finish method on the producer class. Use this method to clean up temporary files or to delete old records.
You will need to register the producer class as a command in a Service.yaml. The naming will be as follows
importer:producer:<importer type name here>
Make sure to use the same name as for the producer class importer type name method.
Note: If you use the importer extension in another extension you will need to add the following configuration to your extension's Services.yaml (just below _defaults):
_instanceof:
Itx\Importer\Consumer\ConsumerInterface:
tags: [ 'queue_consumer' ]
lazy: true
Itx\Importer\Command\Producer\AbstractJobProducer:
tags: [ 'queue_producer' ]
lazy: true
That's it! You can now set up as many queue workers as you want as well as configure a scheduler task for the producer and the import will be executed automatically.
Race conditions and locking
Because every job could be run in parallel by the runtime, there is a chance that the same records might be tried to be imported
multiple times.
To prevent that from happening, you can use the Itx\Importer\Service\LockingService::createLock()
method. This method returns a
Lock object, on which you can call the acquire()
method.
Make sure to call it with the blocking=true
argument. Also make sure to call the release()
method on the lock object when you
are done with it.
The lock is autoconfigured to be released when the process ends, but it's good practice to release it manually.
Child jobs
You can create jobs that are discovered while a job is being processed. This is useful if you have to import hierarchical data.
Statistics
To create statistics for your import, you can use the Itx\Importer\Service\StatisticsService::addStatistic()
method.
The method can run in parallel and will automatically aggregate the statistics for you.
To have data in different places make sure to use the same table name and record name for the same data. The values will be added
together, when called multiple times.
You can see the resulting statistics records in the backend module.
Cleanup
There is a cleanup command that can be used to delete previous imports and their data.
vendor/bin/typo3cms importer:cleanup <keep number of imports>
All versions of importer with dependencies
typo3/cms-core Version ^12
typo3/cms-rte-ckeditor Version ^12
typo3/cms-fluid-styled-content Version ^12
typo3/cms-beuser Version ^12
typo3/cms-backend Version ^12
typo3/cms-scheduler Version ^12
symfony/serializer Version ^6