Libraries tagged by scrawl

Crawl, match, parse IP or IP range, check if IP or range is in another range. Support IPv4, IPv6, IP Interval, Wildcard and CIDR. Check if IP is Cloudflare node IP, Google bot IP. 爬取，正则匹配，解析 IP 和 IP 范围，检测 IP 或范围是否在另一个范围中。支持 IPv4，IPv6，区间、通配符或 CIDR 表示的 IP 范围。检测 IP 是否是 Cloudflare 节点或 Google 漫游器 IP

Go to Download

Helper for sending requests to luka-dev/headless-task-server

Go to Download

A tool to crawl your own laravel installation checking your HTTP status codes

Go to Download

Instrument headless chrome/chromium instances from php5.6

Go to Download

A new generation of multi-process async event-driven spider engine based on Workerman

Go to Download

A crawler to find all unique internal pages on a given website

Go to Download

The BVP Scraper for Boatrace.

Go to Download

DOM crawler with advanced selector API and assertions.

Go to Download

The Google Analytics module consists of 2 components that can be employed independently: The Google Logger injects the google analytics javascript snippet into your source code and logs relevant events (as of now only crawler visits) The Analyzer adds the Google Analytics UI to your CMS.

Go to Download

PHP Sitemap crawler

Go to Download

The Push API allows you to *push* items and security identities, as opposed to letting standard Coveo Cloud V2 crawlers *pull* this data from a content repository. This is especially useful when you need to index content from a cloud or on-premises system for which no dedicated source type exists in the Coveo Cloud V2 platform.

Go to Download

A custom middleware package for Laravel. Temporarily blocks crawlers scanning for vulnerabilities.

Go to Download

QueryList Plugin: Use Puppeteer to crawl Javascript dynamically rendered pages.(Headless Chrome ) 使用Puppeteer采集JavaScript动态渲染的页面

Go to Download

A PHP library used to download, parse and crawl sitemap.xml files.

Go to Download

Magento2: prevent session creation for bots & crawlers.

Go to Download