Libraries tagged by crawlers
bitandblack/sitemap
2394 Downloads
Creates a sitemap.xml by parsing the whole website.
51degrees/fiftyone.devicedetection
8403 Downloads
Device detection engines for 51Degrees Pipeline API. Parse HTTP headers to detect hardware, operating system, browser, and crawler information.
crossjoin/browscap
234335 Downloads
The standalone PHP Browscap parser Crossjoin\Browscap detects browser properties as well as device information based on the user agent string of the requesting browsers and search engines, using the data from the Browser Capabilities Project. It's several hundred times faster than the build-in PHP function get_browser(), and faster than other Browscap PHP libraries, with much lower memory consumption. Optionally Crossjoin\Browscap automatically updates the Browscap data, so you're always up-to-date. The newest version is build for PHP 7.x, for PHP >= 5.6 use version 2.x, for PHP >= 5.3 use version 1.x.
zrashwani/arachnid
20195 Downloads
A crawler to find all unique internal pages on a given website
silverstripe-labs/googleanalytics
10016 Downloads
The Google Analytics module consists of 2 components that can be employed independently: The Google Logger injects the google analytics javascript snippet into your source code and logs relevant events (as of now only crawler visits) The Analyzer adds the Google Analytics UI to your CMS.
s25/dom-trawler
4896 Downloads
Yet another DOM crawler, inspired by symfony/dom-crawler
mjp91/sitemap-generator
15356 Downloads
Generates an XML site map for use by web crawlers
mauricerenck/darkvisitors
48 Downloads
Kirby robots.txt plugin for blocking AI Crawlers and Bots
marcvanh/laravel-bot-block
55 Downloads
A custom middleware package for Laravel. Temporarily blocks crawlers scanning for vulnerabilities.
crawly/html-to-pdf
3305 Downloads
Lib para converter HTML em PDF de uma maneira segura para as necessidades dos crawlers.
alapi/querylist
455 Downloads
Simple, elegant, extensible PHP Web Scraper (crawler/spider),Use the css3 dom selector,Based on phpQuery! 简洁、优雅、可扩展的PHP采集工具(爬虫),基于phpQuery。
codeguy/arachnid
9708 Downloads
A crawler to find all unique internal pages on a given website
zhiyang/php-crawler
14 Downloads
Lightweight web crawler for php
yidas/tw-stock-crawler
34 Downloads
Taiwan Stock Opendata Crawler with realtime - 台灣股市股票資訊擷取器
w3zone/crawler
115 Downloads