Download the PHP package cobaia/marsvin without Composer

Marsvin

What is it?

Have you ever write a crawler or parser?

If yes, you must know that is always a trivial task, but we have always to think how structure our code to do such a thing...

So... to solve that Marvins was created, Marvins provide a simple API and structure to be followed to you create your parsers or crawler. The main focus is to facilitate the task of parser data from external resources, to extract data from websites or import data from XML, CSV files etc...

And it has more, as a plus it comes with a Process management, that makes enable you open more than one PHP process, so you are able to do more than one thing at the same time.

How to use it?

Create a composer.json

Run the command:

Create your console command:

File: console.php

After create the console you are already enable to run the command: php app/console.php

You will check that we have two commands to use:

Marsvin use the following nomenclature:

• Provider: It's the name of the operation that you will be doing, an example of provider would be: Facebook, Github, Google, etc..
• Requester: It's the layer responsible to make the requests of one provider, for example for Facebook would be one HTTP Request, for Github maybe it can be one GIT operation, for another provider it can be one FTP access etc...
• Parser: Once you the request operation has been done, the parser layers comes this layer will take care of the data, so if you want to setup some entities of doctrine, or do you want to create some array of datas, etc... Or you want to normalize some how the data, this is the layer that you will be using to do such a task.
• Persister: Once you have parsed your data, it goes to the Persister layer, here is where you will do forever you want with the data, in doctrine for example you will be able to persist and flush the data into database, or if you want to persists in file system, persists sending one email, or whatever the Persister layer will be handling that to you.

Creating our first provider

To create our provider, marvins has one command that create the folder structure to you:

You will check that Marsvin will generate the following folder tree to you:

Now it's time to setup the adapters, Marsvin using the adapter pattern to define each one of the layers, so for example if you want to make HTTP Request you can setup one HttpAdapter to use in the requester layer.

By default Marvins comes with few adapters:

• Requester: DefaultAdapter.php BuzzAdapter.php
• Parser: DomAdapter.php
• Persister: DefaultAdapter.php DoctrineAdapter.php

So let's setup our provider:

The Requester:

The Parser:

The Persister:

To run the command do the follow:

You can check what is happening here: