Libraries tagged by crawler

szczyglis/php-ultra-small-proxy

17 Favers
226 Downloads

Lightweight proxy with full support for sessions, cookies, POST/FORM submissions, and URL rewriting.

Go to Download


sofyco/spider-bundle

0 Favers
542 Downloads

Symfony Spider Bundle

Go to Download


sofyco/spider

0 Favers
581 Downloads

Crawler + Scraper + Parser library

Go to Download


mantoufan/yzhanip

57 Favers
1262 Downloads

Crawl, match, parse IP or IP range, check if IP or range is in another range. Support IPv4, IPv6, IP Interval, Wildcard and CIDR. Check if IP is Cloudflare node IP, Google bot IP. 爬取,正则匹配,解析 IP 和 IP 范围,检测 IP 或范围是否在另一个范围中。支持 IPv4,IPv6,区间、通配符或 CIDR 表示的 IP 范围。检测 IP 是否是 Cloudflare 节点或 Google 漫游器 IP

Go to Download


itgalaxy/webcrawler-verifier

7 Favers
2947 Downloads

PHP library providing functionality to verify that user-agents are who they claim to be.

Go to Download


haikson/sitemap-generator

36 Favers
0 Downloads

Sitemap crawler and generator class

Go to Download


fkrzski/robots-txt

4 Favers
858 Downloads

A modern, fluent PHP package for managing robots.txt rules with type safety and great developer experience

Go to Download


bitandblack/sitemap

0 Favers
3503 Downloads

Creates a sitemap.xml by parsing the whole website.

Go to Download


godbout/htmlpagedom

1 Favers
26915 Downloads

jQuery-inspired DOM manipulation extension for Symfony's Crawler

Go to Download


crossjoin/browscap

45 Favers
234370 Downloads

The standalone PHP Browscap parser Crossjoin\Browscap detects browser properties as well as device information based on the user agent string of the requesting browsers and search engines, using the data from the Browser Capabilities Project. It's several hundred times faster than the build-in PHP function get_browser(), and faster than other Browscap PHP libraries, with much lower memory consumption. Optionally Crossjoin\Browscap automatically updates the Browscap data, so you're always up-to-date. The newest version is build for PHP 7.x, for PHP >= 5.6 use version 2.x, for PHP >= 5.3 use version 1.x.

Go to Download


bringyourownideas/laravel-sitemap

9 Favers
7867 Downloads

A simple website crawler & sitemap generator without a headless browser for Laravel 5.8+

Go to Download


yannkuesthardt/laravel-spamprotect

2 Favers
525 Downloads

Encrypt e-mail addresses and phone numbers to protect against unwanted crawlers & spam.

Go to Download


silverstripe-labs/googleanalytics

32 Favers
10122 Downloads

The Google Analytics module consists of 2 components that can be employed independently: The Google Logger injects the google analytics javascript snippet into your source code and logs relevant events (as of now only crawler visits) The Analyzer adds the Google Analytics UI to your CMS.

Go to Download


orangecat/checkip

8 Favers
149 Downloads

Prevent crawlers and IPs reported from creating a session

Go to Download


mjp91/sitemap-generator

1 Favers
17013 Downloads

Generates an XML site map for use by web crawlers

Go to Download


<< Previous Next >>