Libraries tagged by crawl

Integration with prerender.io service

Go to Download

Translations Manager extension to use Google translation crawling to auto translate your __(), trans() fn

Go to Download

Lightweight proxy with full support for sessions, cookies, POST/FORM submissions, and URL rewriting.

Go to Download

Tiny WebCrawler for Laravel using Playwright

Go to Download

Find dead links, 404, 301 and more (command line or admin).

Go to Download

A SilverStripe module to optimise the Meta, crawling, indexing, and sharing of your website content (forked from Cyber-Duck/Silverstripe-SEO)

Go to Download

A tool to crawl your own laravel installation checking your HTTP status codes

Go to Download

Beanbun 是用 PHP 编写的多进程网络爬虫框架，具有良好的开放性、高可扩展性

Go to Download

A SilverStripe module to optimise the Meta, crawling, indexing, and sharing of your website content

Go to Download

A new generation of multi-process async event-driven spider engine based on Workerman

Go to Download

A powerful web crawler and web scraper with Blackfire support

Go to Download

High-performance device detection engines for the 51Degrees Pipeline API. Accurately identify devices, operating systems, browsers, and crawlers from HTTP headers.

Go to Download

The standalone PHP Browscap parser Crossjoin\Browscap detects browser properties as well as device information based on the user agent string of the requesting browsers and search engines, using the data from the Browser Capabilities Project. It's several hundred times faster than the build-in PHP function get_browser(), and faster than other Browscap PHP libraries, with much lower memory consumption. Optionally Crossjoin\Browscap automatically updates the Browscap data, so you're always up-to-date. The newest version is build for PHP 7.x, for PHP >= 5.6 use version 2.x, for PHP >= 5.3 use version 1.x.

Go to Download

A fast, simple and powerful PHP web crawler (scraper/spider) 快速、简洁且强大的爬虫/采集框架

Go to Download

The Google Analytics module consists of 2 components that can be employed independently: The Google Logger injects the google analytics javascript snippet into your source code and logs relevant events (as of now only crawler visits) The Analyzer adds the Google Analytics UI to your CMS.

Go to Download