Libraries tagged by scrawl

A simple website crawler & sitemap generator without a headless browser for Laravel 5.8+

Go to Download

PHPCreeper plugin for webman

Go to Download

Creates a sitemap.xml by parsing the whole website.

Go to Download

Device detection engines for 51Degrees Pipeline API. Parse HTTP headers to detect hardware, operating system, browser, and crawler information.

Go to Download

jQuery-inspired DOM manipulation extension for Symfony's Crawler

Go to Download

The standalone PHP Browscap parser Crossjoin\Browscap detects browser properties as well as device information based on the user agent string of the requesting browsers and search engines, using the data from the Browser Capabilities Project. It's several hundred times faster than the build-in PHP function get_browser(), and faster than other Browscap PHP libraries, with much lower memory consumption. Optionally Crossjoin\Browscap automatically updates the Browscap data, so you're always up-to-date. The newest version is build for PHP 7.x, for PHP >= 5.6 use version 2.x, for PHP >= 5.3 use version 1.x.

Go to Download

Encrypt e-mail addresses and phone numbers to protect against unwanted crawlers & spam.

Go to Download

NeosCMS package that is able to crawl the whole page and check the links for broken links

Go to Download

Parallel URL crawling extension to curl-easy.

Go to Download

Yet another DOM crawler, inspired by symfony/dom-crawler

Go to Download

Simple, elegant, extensible PHP Web Scraper (crawler/spider),Use the css3 dom selector,Based on phpQuery! 简洁、优雅、可扩展的PHP采集工具(爬虫)，基于phpQuery。

Go to Download

Prevent crawlers and IPs reported from creating a session

Go to Download

Crawl the web with kohana or phantomjs.

Go to Download

Display fully customizable, responsive, and search engine crawlable Twitter feeds on your TYPO3 website. The TYPO3 Twitter Extension easily integrates to display Tweets that perfectly match your site's appearance and offers many customization options. Easily embed Twitter Timelines, Feeds, Follow buttons, and Like buttons on posts, pages, and widget areas. Documentation & Free Support: https://t3planet.com/typo3-twitter-extension

Go to Download

Generates an XML site map for use by web crawlers

Go to Download