Libraries tagged by crawler

proxycrawl/proxycrawl

19 Favers
85981 Downloads

A lightweight, dependency free PHP class that acts as wrapper for ProxyCrawl API

jyggen/curl

80 Favers
146134 Downloads

A simple and lightweight cURL library with support for asynchronous requests.

vipnytt/robotstxtparser

27 Favers
694805 Downloads

Robots.txt parsing library, with full support for every directive and specification.

smochin/instagram-php-crawler

46 Favers
8374 Downloads

A simple PHP Crawler for Instagram

friends-of-hyva/magento2-crawler-session

13 Favers
3678 Downloads

Prevent crawlers from creating a session

dachcom-digital/dynamic-search-data-provider-crawler

8 Favers
25605 Downloads

aoepeople/crawler

58 Favers
287765 Downloads

Crawler extension for TYPO3

baba/sitemap-crawler

Favers
Downloads

vipnytt/useragentparser

2 Favers
871517 Downloads

User-Agent parser for robot rule sets

tomverran/robots-txt-checker

13 Favers
53531 Downloads

Given a robots.txt file, user agent and URL path will tell you whether you're allowed to access a page

sleeping-owl/apist

312 Favers
5014 Downloads

Package to provide api-like access to foreign sites based on html parsing

opensearchserver/opensearchserver

52 Favers
65430 Downloads

PHP library for OpenSearchServer: professionnal search engine, crawlers (web, file, database), REST APIs, .... This library uses OpenSearchServer's V2 API.

eddieace/php-simple

13 Favers
54289 Downloads

crwlr/robots-txt

12 Favers
16981 Downloads

Robots Exclusion Standard/Protocol Parser for Web Crawling/Scraping

bvp/boatrace-scraper

1 Favers
10533 Downloads

The BVP Boatrace Scraper package for Boatrace.

<< Previous Next >>