Download the PHP package monteduro/laravel-datafast-aicrawl without Composer

On this page you can find all versions of the php package monteduro/laravel-datafast-aicrawl. It is possible to download/install these versions without Composer. Possible dependencies are resolved automatically.

FAQ

After the download, you have to make one include require_once('vendor/autoload.php');. After that you have to import the classes with use statements.

Example:
If you use only one package a project is not needed. But if you use more then one package, without a project it is not possible to import the classes with use statements.

In general, it is recommended to use always a project to download your libraries. In an application normally there is more than one library needed.
Some PHP packages are not free to download and because of that hosted in private repositories. In this case some credentials are needed to access such packages. Please use the auth.json textarea to insert credentials, if a package is coming from a private repository. You can look here for more information.

  • Some hosting areas are not accessible by a terminal or SSH. Then it is not possible to use Composer.
  • To use Composer is sometimes complicated. Especially for beginners.
  • Composer needs much resources. Sometimes they are not available on a simple webspace.
  • If you are using private repositories you don't need to share your credentials. You can set up everything on our site and then you provide a simple download link to your team member.
  • Simplify your Composer build process. Use our own command line tool to download the vendor folder as binary. This makes your build process faster and you don't need to expose your credentials for private repositories.
Please rate this library. Is it a good library?

Example code of monteduro/laravel-datafast-aicrawl

Informations about the package laravel-datafast-aicrawl

Laravel DataFast AI Crawl

Laravel DataFast AI Crawl — bot traffic tracking

Latest Version on Packagist Total Downloads

Server-side AI crawler & bot traffic tracking for Laravel, reporting to DataFast.

See when AI assistants (ChatGPT, Claude, Perplexity…), search crawlers (Googlebot, Bingbot…) and model-training bots (GPTBot, ClaudeBot, CCBot…) request pages on your site. These bots fetch raw HTML and skip frontend JavaScript, so DataFast's browser script never sees them — this package detects them in your backend.

Unofficial PHP port of @datafast/ai-crawl. Not affiliated with DataFast. The ingestion endpoint it posts to is undocumented and may change.

Install

The service provider is auto-discovered and the middleware is auto-registered on the web group. Two things to set in .env:

That's the whole setup. Requires PHP 8.2+ and Laravel 11/12/13.

About the queue

Every detected crawler request triggers one outbound HTTP POST to DataFast. That call is dispatched as a best-effort queued job (no retries, errors swallowed) and the middleware runs after the response is sent (terminate()), so even on the sync driver the visitor is never blocked. Point DATAFAST_AICRAWL_CONNECTION/QUEUE at a real worker for best throughput.

How it works

For each GET/HEAD request the middleware classifies the User-Agent against a built-in catalog of ~80 crawlers, skips non-page traffic (static extensions, /api, /_next, Sec-Fetch-Dest sub-resources, …), and if it's a real crawler page hit, queues the event to DataFast.

Payload sent: websiteId, domain, href, referrer, and an ai object with provider, agent, category (answer_fetch / search_index / training / ai_crawler), userAgent, ip, statusCode.

Configuration

Optional — publish only to customize defaults:

Env var Default Purpose
DATAFAST_WEBSITE_ID — Required. Your DataFast website id.
DATAFAST_AICRAWL_QUEUE default Queue name for the tracking job.
DATAFAST_AICRAWL_CONNECTION default Queue connection.
DATAFAST_AICRAWL_ENABLED true Master on/off switch.
DATAFAST_AICRAWL_ENDPOINT https://datafa.st/api/ai-crawls Ingestion endpoint.
DATAFAST_AICRAWL_DOMAIN request host Override reported domain.
DATAFAST_AICRAWL_TIMEOUT 5 HTTP timeout (seconds).
DATAFAST_AICRAWL_AUTO_MIDDLEWARE true Auto-append middleware to a group.
DATAFAST_AICRAWL_MIDDLEWARE_GROUP web Group to append the middleware to.

Disable crawler categories or tweak ignored paths/extensions in config/datafast-aicrawl.php.

To register the middleware manually instead, set DATAFAST_AICRAWL_AUTO_MIDDLEWARE=false and attach \Monteduro\DataFastAiCrawl\Http\Middleware\TrackAiCrawler::class yourself.

Keeping the crawler list fresh

The catalog data lives in resources/crawlers.php, generated from the upstream npm bundle (not fetched at runtime). You normally never touch it — a GitHub Action runs weekly, regenerates it from @datafast/ai-crawl, runs the tests, and opens a PR when something changed. Merging that PR auto-tags the next patch version, which Packagist publishes.

To run the sync manually:

The weekly workflow needs Settings → Actions → General → "Allow GitHub Actions to create and approve pull requests" enabled on the repo.

Testing

Credits & License

Crawler catalog and detection logic ported from @datafast/ai-crawl (MIT). MIT licensed — see LICENSE.

All versions of laravel-datafast-aicrawl with dependencies

PHP Build Version
Package Version
Requires php Version ^8.2
illuminate/contracts Version ^11.0|^12.0|^13.0
illuminate/http Version ^11.0|^12.0|^13.0
illuminate/support Version ^11.0|^12.0|^13.0
Composer command for our command line client (download client) This client runs in each environment. You don't need a specific PHP version etc. The first 20 API calls are free. Standard composer command

The package monteduro/laravel-datafast-aicrawl contains the following files

Loading the files please wait ...