Libraries tagged by crawl

The standalone PHP Browscap parser Crossjoin\Browscap detects browser properties as well as device information based on the user agent string of the requesting browsers and search engines, using the data from the Browser Capabilities Project. It's several hundred times faster than the build-in PHP function get_browser(), and faster than other Browscap PHP libraries, with much lower memory consumption. Optionally Crossjoin\Browscap automatically updates the Browscap data, so you're always up-to-date. The newest version is build for PHP 7.x, for PHP >= 5.6 use version 2.x, for PHP >= 5.3 use version 1.x.

Go to Download

A fast, simple and powerful PHP web crawler (scraper/spider) 快速、简洁且强大的爬虫/采集框架

Go to Download

PHP Sitemap crawler

Go to Download

Display fully customizable, responsive, and search engine crawlable Twitter feeds on your TYPO3 website. The TYPO3 Twitter Extension easily integrates to display Tweets that perfectly match your site's appearance and offers many customization options. Easily embed Twitter Timelines, Feeds, Follow buttons, and Like buttons on posts, pages, and widget areas. Documentation & Free Support: https://t3planet.com/typo3-twitter-extension

Go to Download

Kirby robots.txt plugin for blocking AI Crawlers and Bots

Go to Download

A PHP library used to download, parse and crawl sitemap.xml files.

Go to Download

Utilities that are needed in multiple crawler packages.

Go to Download

Contao bundle helper class to detect search engines, bots, spiders, crawlers ...

Go to Download

宇润爬虫框架(Yurun Crawler) 是一个低代码、高性能、分布式爬虫采集框架，这可能是最一把梭的爬虫框架。

Go to Download

【正方教务】爬虫，支持成绩查询、考试查询、课表查询、四六级成绩查询、四六级报名、选课查询、修改密码、获取用户菜单等功能，并且解析数据成易读格式，符合 psr 规范，拿来即用

Go to Download

Crawls Website and can generate backstop json

Go to Download

A versatile and extendable crawler for TYPO3 CMS

Go to Download

This is my package laravel-blade-crawler-detect

Go to Download

Crawls all unique links.

Go to Download

Provides a way to create tests that crawl a site

Go to Download