Libraries tagged by crawler

Laravel middleware za blokiranje AI crawlera i tražilica. Auto-registracija, konfigurabilan, podržava Laravel 10/11/12/13.

Go to Download

Crawl, match, parse IP or IP range, check if IP or range is in another range. Support IPv4, IPv6, IP Interval, Wildcard and CIDR. Check if IP is Cloudflare node IP, Google bot IP. 爬取，正则匹配，解析 IP 和 IP 范围，检测 IP 或范围是否在另一个范围中。支持 IPv4，IPv6，区间、通配符或 CIDR 表示的 IP 范围。检测 IP 是否是 Cloudflare 节点或 Google 漫游器 IP

Go to Download

A professional, production-ready PHP XML sitemap generator with advanced features: concurrency, robots.txt, caching, database storage, change detection, SEO analysis, JavaScript rendering, proxy support, webhooks, and more.

Go to Download

Google search results crawler, get google search results that you need.

Go to Download

Sitemap crawler and generator class

Go to Download

Package to provide api-like access to foreign sites based on html parsing (fork of sleeping-owl/apist package with updated dependencies)

Go to Download

Crawlzone is a fast asynchronous internet crawling framework aiming to provide open source web search and testing solution. It can be used for a wide range of purposes, from extracting and indexing structured data to monitoring and automated testing.

Go to Download

PHPCreeper plugin for webman

Go to Download

High-performance device detection engines for the 51Degrees Pipeline API. Accurately identify devices, operating systems, browsers, and crawlers from HTTP headers.

Go to Download

Last.fm webservice client for php.

Go to Download

A simple website crawler & sitemap generator without a headless browser for Laravel 5.8+

Go to Download

A crawler to find all unique internal pages on a given website

Go to Download

Laravel package for smking AEO — auto-inject JSON-LD, FAQ, and AI summary into Laravel responses so AI crawlers (ChatGPT, Perplexity, Google AI) can cite your pages.

Go to Download

Open-core SEO infrastructure for Laravel: metadata, canonical URLs, social cards, linked JSON-LD, XML sitemaps and crawler controls - the free MIT core.

Go to Download

Generates an XML site map for use by web crawlers

Go to Download