Libraries tagged by crawlers
51degrees/fiftyone.devicedetection
11614 Downloads
Device detection engines for 51Degrees Pipeline API. Parse HTTP headers to detect hardware, operating system, browser, and crawler information.
spatie/laravel-link-checker
52749 Downloads
Check all links in a Laravel app
godbout/htmlpagedom
26858 Downloads
jQuery-inspired DOM manipulation extension for Symfony's Crawler
zrashwani/arachnid
20204 Downloads
A crawler to find all unique internal pages on a given website
zenstruck/dom
2813 Downloads
DOM crawler with advanced selector API and assertions.
xcrawler/xcrawler
880 Downloads
A fast, simple and powerful PHP web crawler (scraper/spider) 快速、简洁且强大的爬虫/采集框架
octopoda/octopus
4403 Downloads
PHP Sitemap crawler
marcvanh/laravel-bot-block
1318 Downloads
A custom middleware package for Laravel. Temporarily blocks crawlers scanning for vulnerabilities.
crwlr/utils
11785 Downloads
Utilities that are needed in multiple crawler packages.
bugbuster/contao-botdetection-bundle
38707 Downloads
Contao bundle helper class to detect search engines, bots, spiders, crawlers ...
yurunsoft/crawler
82 Downloads
宇润爬虫框架(Yurun Crawler) 是一个低代码、高性能、分布式爬虫采集框架，这可能是最一把梭的爬虫框架。
yuan1994/z-crawler
53 Downloads
【正方教务】爬虫，支持成绩查询、考试查询、课表查询、四六级成绩查询、四六级报名、选课查询、修改密码、获取用户菜单等功能，并且解析数据成易读格式，符合 psr 规范，拿来即用
webprofil/crawler
471 Downloads
Crawls Website and can generate backstop json
survos/crawler-bundle
3636 Downloads
Provides a way to create tests that crawl a site
shel/crawler
1865 Downloads
Allows crawling of sitemaps and node-trees