Download the PHP package rgilyov/laravel-csv-importer without Composer

laravel-csv-importer

Flexible and reliable way to import, parse, validate and transform your csv files with laravel

Installation

Register \RGilyov\CsvImporter\CsvImporterServiceProvider inside config/app.php

After installation you may publish default configuration file

Works with laravel 5 and above, hhvm are supported.

Requirements

Each created importer will have built in mutex functionality to make imports secure and avoid possible data incompatibilities, it's important especially in cases when > 100k lines csv files will be imported, due to that a laravel application should have file, redis or memcached cache driver set in the .env file

Basic usage

To create new csv importer, a class should extends \RGilyov\CsvImporter\BaseCsvImporter abstract class or the php artisan make:csv-importer MyImporter console command can be used, after execution new file with name MyImporter.php and basic importer set up will be placed inside app/CsvImporters/ directory.

There are 3 main methods:

• csvConfigurations which returns configurations for the given type of csv, configurations has 3 parts:

  • 'mappings': you may specify fields(headers) which you expect the given csv has and attach rules to each field(header), there are 3 types of rules(filters) which you can specify:

    • To make a field(header) mandatory for the import you need to set required parameter for the field(header) 'name' => ['required'], so if the given csv file won't have the field(header) name an error will be thrown.

    • You may set validation to each field(header), the importer uses laravel validation, so you can use any rules from there https://laravel.com/docs/5.4/validation#available-validation-rules to check csv values 'email' => ['required', validation => ['email']], so if a value won't be a valid email, the csv line which contains the email will be put inside invalid($item) method otherwise inside handle($item).

    • You can cast csv values in any native php type and format date which csv contains 'name' => ['required', 'cast' => 'string'] or 'birth_date' => ['cast' => ['string', 'date']] cast will work before validation.

  • 'csv_files': files specified inside the key will be created, you can write csv lines inside each file, with $this->insertTo('csv_file_name', $item); method, for example you can separate invalid csv lines from valid, it uses laravel filesystem \Storage support class https://laravel.com/docs/5.4/filesystem.

  • 'config': you may overwrite global config/csv-importer.php configurations hear for the given csv importer.
• handle: will be executed for csv lines which passed validation.
• invalid: will be executed for csv lines which didn't pass validation.

Let's finally import a csv:

After the import had started you won't be able to start another import until the first one finished.

During the import though you may want to know progress of the running process:

At the end of the import you will have key finished => true inside meta data. So you will need to finish your csv import:

If something went wrong, you can cancel the current import process:

Importer customization

Besides methods above the importer also has a list of methods which can help you to easily expand your functionality for some particular cases:

• before - will be executed before start of an import process
• after - will be executed after an import process finished
• onCancel - will be executed before abortion of an import process
• initProgressBar - will initialize new progress bar
• progressBarDetails - additional information for progress bar
• setFinalDetails - set additional information after finish() of an import process
• setError - add an error to the list of errors which will be thrown (if exists)

Basic csv aggregations

If a csv file is set to an import class you can count it, get distinct values from it or loop through the csv:

All methods above returns false if a csv file wasn't set.

Configurations

There are 3 layers of configurations:

• 1) global config/csv-importer.php configuration file, which has default parameters applied for all csv importers
• 2) local configurations, which csvConfigurations() method returns, overwrites global configurations
• 3) manual configuration customization with setters, overwrites global and local configurations

1) Global configurations:

2) Local configurations:

3) Configurations with setters:

From csv headers to defined mappings and reverse array transformation

Sounds awful, better just show how it works:

Suppose we have a csv file with this structure:

And we make an import class and define mappings, in this case we interested only in name field(header):

Mutex key concatenation

There are cases when you need to be able to run several similar import processes at the same time, for example you have guitars and guitar_companies tables in your db and two csv files ltd_guitars.csv and black_machine_guitars.csv and you use the same import class for both csvs, but since import process is locked you not able to import both at the same time, in this case use mutex key concatenation, to have different mutex key for each guitar company:

Now you can run the importer for each company in the same time. But not for the same company.

Custom filters

As mentioned above in the Basic usage chapter, the csv importer has 3 types of filters, which you can specify for each csv field(header), but some times you need to do something more sophisticated, for example:

• if a given csv has field(header) A OR field(header) B and if both are missing throw headers validation error, in this case parameter required will be insufficient for the task due to the csv import will check each field(header) which has the parameter which is AND logic, in this case you need to create headers filter.

• Or for example you need to make white list check for all values in a particular field(header), which is new validation rule(filter).

• Or you may want to perform some advanced transformation for csv values, in this situation you will need to create cast filter.

Custom headers filters

To make custom headers filter you need to make a class which will extends \RGilyov\CsvImporter\BaseHeadersFilter or just run php artisan make:csv-importer-headers-filter MyHeadersFilter which will make MyHeadersFilter.php file with basic set up inside app/CsvImporters/HeadersFilters/ folder:

The filter has property errorMessage where you can specify error message which will be thrown if the method filter will return false, after start of an import process. You may also specify name of the filter to return it, or check if it exists, or unset it and etc, to do so you need to specify protected $name property and set whatever name you want, short name of the class will be by default, in this case MyHeadersFilter

To register your new header filter for a importer you need to use addHeadersFilter() static function:

Of course you can flush, get, unset and check filters during an import process:

Custom validation filters

To make custom validation filter you need to make a class which will extends \RGilyov\CsvImporter\BaseValidationFilter or just run php artisan make:csv-importer-validation-filter MyValidationFilter which will make MyValidationFilter.php file with basic set up inside app/CsvImporters/ValidationFilters/ folder:

If for headers filters property name not so important, for validation filters it's quit useful to have specified because to use it, we will need to set it inside validation property which contains in csvConfigurations() method for a field(header): 'name' => ['validation' => 'bad_word_validation']

But there are cases when you'd like to make a global sort of speak validation filter, to be able validate whole csv entity array, for example a csv line is valid only if it has not empty user name or not empty user first name and last name, to get array with all csv columns instead of just a value you need to set public $global = true, of course no need to specify such validation filter inside an import class csv configurations

All filters manipulation is similar to what was described in Custom headers filters chapter:

Custom cast filters

To make custom cast filter you need to make a class which will extends \RGilyov\CsvImporter\BaseCastFilter or just run php artisan make:csv-importer-cast-filter MyCastFilter which will make MyCastFilter.php file with basic set up inside app/CsvImporters/CastFilters/ folder:

name as important as for validation filters, due to we will set it inside csvConfigurations() mappings, for a field (header): 'field' => ['cast' => 'lowercase']

All filters manipulation is similar to what was described in Custom headers filters and Custom validation filters chapters:

The best way to register your custom filters

I think the best way to register custom filters is to use a service provider: https://laravel.com/docs/5.4/providers