Libraries tagged by crawl
vlados/laravel-blade-crawler-detect
5788 Downloads
Boost Lighthouse and PageSpeed scores by hiding cookie banners, chat widgets and heavy scripts from crawlers via two Blade directives. Octane-safe.
umpirsky/centipede-crawler
454 Downloads
Crawls all unique links.
survos/crawler-bundle
5394 Downloads
Provides a way to create tests that crawl a site
shipwake/laravel-ai-crawl
194 Downloads
Server-side AI crawler tracking middleware for Laravel applications.
piedweb/crawler
379 Downloads
Web Crawler to check few SEO basics.
luyadev/luya-module-crawler
33998 Downloads
An full search page crawler to enable complex and customized searching abilities.
leowebguy/craft-crawler-detect
6080 Downloads
Craft plugin for detecting bots/crawlers/spiders
kaishiyoku/hera-rss-crawler
4557 Downloads
Modern library to handle RSS/Atom feeds
jacyimp/crawler-verifier
207 Downloads
Verify that web crawlers are actually who they claim to be.
hirak/packagist-crawler
15 Downloads
build mirror of packagist.org
darvinstudio/darvin-crawler-bundle
1441 Downloads
This bundle provides console command that detects broken links on your website.
rfussien/leboncoin-crawler
360 Downloads
Makes data extraction from leboncoin.fr easy
hedii/php-crawler
1097 Downloads
A crawler application written with php and Laravel that finds email addresses on the internets.
codeguy/arachnid
9790 Downloads
A crawler to find all unique internal pages on a given website
web-nomads/wn-ai-bridge
53 Downloads
TYPO3 extension for generating llms.txt links based on the llmstxt.org specification and an on-site AI search assistant that helps visitors find information via ke_search and indexed_search.