Libraries tagged by scrawl

This repository contains a list of of HTTP user-agents used by robots, crawlers, and spiders as in single JSON file.

Go to Download

Robots.txt parsing library, with full support for every directive and specification.

Go to Download

A configurable and extensible PHP web spider

Go to Download

A web crawler or spider library based on Symfony components

Go to Download

PHPScraper, built with simplicity in mind. See tests/ for more examples.

Go to Download

A site search engine

Go to Download

cURL wrapper for PHP. Supports parallel and non-blocking requests. For high speed crawling, see stil/curl-robot.

Go to Download

Web crawling and scraping library.

Go to Download

A lightweight, dependency free PHP class that acts as wrapper for Crawlbase API

Go to Download

CLI tool to crawl a website and check HTTP status code

Go to Download

A lightweight, dependency free PHP class that acts as wrapper for ProxyCrawl API

Go to Download

Warming - Warms up Frontend caches based on an XML sitemap. Cache warmup can be triggered via TYPO3 backend or using a console command. Supports multiple languages and custom crawler implementations.

Go to Download

Go to Download

A simple PHP Crawler for Instagram

Go to Download

A Symfony bundle for the Crawler-Detect library (detects bots/crawlers/spiders via the user agent)

Go to Download