Libraries tagged by crawlora

PHPScraper, built with simplicity in mind. See tests/ for more examples.

Go to Download

A web crawler or spider library based on Symfony components

Go to Download

A Symfony bundle for the Crawler-Detect library (detects bots/crawlers/spiders via the user agent)

Go to Download

A lightweight, dependency free PHP class that acts as wrapper for Crawlbase API

Go to Download

Robots.txt parsing library, with full support for every directive and specification.

Go to Download

CLI tool to crawl a website and check HTTP status code

Go to Download

A lightweight, dependency free PHP class that acts as wrapper for ProxyCrawl API

Go to Download

Warming - Warms up Frontend caches based on an XML sitemap. Cache warmup can be triggered via TYPO3 backend or using a console command. Supports multiple languages and custom crawler implementations.

Go to Download

Go to Download

A simple PHP Crawler for Instagram

Go to Download

Prevent crawlers from creating a session

Go to Download

Crawler extension for TYPO3

Go to Download

User-Agent parser for robot rule sets

Go to Download

Given a robots.txt file, user agent and URL path will tell you whether you're allowed to access a page

Go to Download

PHP library for OpenSearchServer: professionnal search engine, crawlers (web, file, database), REST APIs, .... This library uses OpenSearchServer's V2 API.

Go to Download