Libraries tagged by crawler

Package to provide api-like access to foreign sites based on html parsing (fork of sleeping-owl/apist package with updated dependencies)

Go to Download

Bot and AI-crawler control presets for the Phirewall PHP firewall - block AI scrapers and rate-limit aggressive SEO crawlers

Go to Download

A Laravel package to fetch webpage metadata ( Open Graph | Twitter | Facebook | Article ) ect...

Go to Download

PHPCreeper plugin for webman

Go to Download

Creates a sitemap.xml by parsing the whole website.

Go to Download

High-performance device detection engines for the 51Degrees Pipeline API. Accurately identify devices, operating systems, browsers, and crawlers from HTTP headers.

Go to Download

jQuery-inspired DOM manipulation extension for Symfony's Crawler

Go to Download

Last.fm webservice client for php.

Go to Download

A simple website crawler & sitemap generator without a headless browser for Laravel 5.8+

Go to Download

Encrypt e-mail addresses and phone numbers to protect against unwanted crawlers & spam.

Go to Download

Laravel package for smking AEO — auto-inject JSON-LD, FAQ, and AI summary into Laravel responses so AI crawlers (ChatGPT, Perplexity, Google AI) can cite your pages.

Go to Download

Yet another DOM crawler, inspired by symfony/dom-crawler

Go to Download

Laravel SEO package for metadata, canonical URLs, Open Graph, linked JSON-LD, XML sitemaps and crawler controls - free and MIT licensed.

Go to Download

Generates an XML site map for use by web crawlers

Go to Download

Kirby robots.txt plugin for blocking AI Crawlers and Bots

Go to Download