Libraries tagged by craw

PHPCreeper plugin for webman

Go to Download

Creates a sitemap.xml by parsing the whole website.

Go to Download

Device detection engines for 51Degrees Pipeline API. Parse HTTP headers to detect hardware, operating system, browser, and crawler information.

Go to Download

The standalone PHP Browscap parser Crossjoin\Browscap detects browser properties as well as device information based on the user agent string of the requesting browsers and search engines, using the data from the Browser Capabilities Project. It's several hundred times faster than the build-in PHP function get_browser(), and faster than other Browscap PHP libraries, with much lower memory consumption. Optionally Crossjoin\Browscap automatically updates the Browscap data, so you're always up-to-date. The newest version is build for PHP 7.x, for PHP >= 5.6 use version 2.x, for PHP >= 5.3 use version 1.x.

Go to Download

A crawler to find all unique internal pages on a given website

Go to Download

Lib de validação de renavam no laravel

Go to Download

Lib de validação de inscrição estadual no laravel

Go to Download

Lib para converter HTML em PDF de uma maneira segura para as necessidades dos crawlers.

Go to Download

Lib de validação de cei no laravel

Go to Download

A library meant to assist with breaking captchas, using various providers

Go to Download

Go to Download

Parallel URL crawling extension to curl-easy.

Go to Download

The Google Analytics module consists of 2 components that can be employed independently: The Google Logger injects the google analytics javascript snippet into your source code and logs relevant events (as of now only crawler visits) The Analyzer adds the Google Analytics UI to your CMS.

Go to Download

Yet another DOM crawler, inspired by symfony/dom-crawler

Go to Download

Crawl the web with kohana or phantomjs.

Go to Download