Download the PHP package crwlr/crwl-extension-utils without Composer

crwl.io Extension Utils

These utilities enable you to develop extensions for the crwl.io web crawling and scraping app.

How to Package Your Custom Steps for the crwl.io App

The crwlr.software documentation provides a detailed explanation of how to build your own steps for the crwlr/crawler library.

Once you've successfully created a working step, the most challenging part of your task is already complete. To integrate your custom steps into the crwl.io app, follow these simple steps:

• Package it as a Composer package in a GitHub repository.
• Create a StepBuilder
• Create a laravel ServiceProvider and register your extension steps using the ExtensionPackageManager included in this package.

Composer Package Setup

Ensure your package repository's composer.json file resembles the following template:

Feel free to customize it according to your preferences, but ensure that it includes dependencies on the following three packages: crwlr/crawler, crwlr/crwl-extension-utils and illuminate/support.

For a well-organized project structure, we recommend the following folder arrangement:

In your .gitignore, include at least the following entries:

Step Builders

To create a StepBuilder for your step, follow the example below:

If your step requires configuration, define the necessary parameters in the configParams() method. The crwl.io app's crawler creation/editing form will display corresponding inputs for these config options. When the crawler runs, the StepBuilder::configToStep() method is invoked with the user-saved configuration data. In this method, construct your custom step with the configured values and return it.

Currently, the available config param types are string, int, and bool. Optionally, you can specify a default value (default()), an input label (inputLabel()), and a description text (description()) for each configuration parameter.

If your step doesn't require any settings, the StepBuilder looks rather minimalistic:

If your step needs a filesystem path, where it can store files, you can use $this->fileStoragePath inside the builder. The crwl.io app sets this path for all step builders before building any steps.

When creating a step builder for an HTTP loading step, please implement the following method:

ServiceProvider and Registering Package and Steps

To make your steps accessible in the crwl.io app, the final step is to register an extension package and all your steps using the ExtensionPackageManager included in this package. Since crwl.io is a Laravel application, this is accomplished through a ServiceProvider class:

To complete the setup, add the ServiceProvider to the extra section in the composer.json file:

With these configurations in place, your extension package is ready for use. If your extension is private, ensure you grant access to the crwlrsoft GitHub organization. As a super-user on your crwl.io instance, you can then install your extension via the extensions page in the app.

Custom Steps Performing HTTP Requests Without the crwlr/crawler HttpLoader

In scenarios where your custom steps need to execute HTTP requests that cannot leverage the HttpLoader from the crwlr/crawler package—such as when utilizing a REST API SDK to retrieve data from an API—you'll need to ensure that every HTTP request is tracked when executing your custom steps within the crwl.io app.

To accomplish this, you have two options:

• Use a Guzzle Client instance generated by the TrackingGuzzleClientFactory.
• Alternatively, manually invoke the trackHttpResponse() or trackHeadlessBrowserResponse() methods of the RequestTracker following each request.

Using a guzzle Client instance

If you want to use a Guzzle Client instance for the requests (it's common practice for PHP API SDKs to let you provide your own Guzzle instance), use the TrackingGuzzleClientFactory from this package:

You can also pass your custom Guzzle configuration as an argument:

Using the RequestTracker

In scenarios where utilizing a Guzzle Client instance for requests is not feasible, you need to call either RequestTracker::trackHttpResponse() or if your request was executed using a headless browser RequestTracker::trackHeadlessBrowserResponse().

If you can provide request/response instances implementing the PSR-7 RequestInterface and/or ResponseInterface, please do so: