Libraries tagged by crawler
blackfire/player
16283 Downloads
A powerful web crawler and web scraper with Blackfire support
51degrees/fiftyone.devicedetection
11260 Downloads
Device detection engines for 51Degrees Pipeline API. Parse HTTP headers to detect hardware, operating system, browser, and crawler information.
godbout/htmlpagedom
26538 Downloads
jQuery-inspired DOM manipulation extension for Symfony's Crawler
zenstruck/dom
2101 Downloads
DOM crawler with advanced selector API and assertions.
xcrawler/xcrawler
879 Downloads
A fast, simple and powerful PHP web crawler (scraper/spider) 快速、简洁且强大的爬虫/采集框架
orangecat/checkip
106 Downloads
Prevent crawlers and IPs reported from creating a session
octopoda/octopus
4315 Downloads
PHP Sitemap crawler
marcvanh/laravel-bot-block
1136 Downloads
A custom middleware package for Laravel. Temporarily blocks crawlers scanning for vulnerabilities.
crwlr/utils
11481 Downloads
Utilities that are needed in multiple crawler packages.
bugbuster/contao-botdetection-bundle
37998 Downloads
Contao bundle helper class to detect search engines, bots, spiders, crawlers ...
codeguy/arachnid
9735 Downloads
A crawler to find all unique internal pages on a given website
yurunsoft/crawler
80 Downloads
宇润爬虫框架(Yurun Crawler) 是一个低代码、高性能、分布式爬虫采集框架，这可能是最一把梭的爬虫框架。
yuan1994/z-crawler
52 Downloads
【正方教务】爬虫，支持成绩查询、考试查询、课表查询、四六级成绩查询、四六级报名、选课查询、修改密码、获取用户菜单等功能，并且解析数据成易读格式，符合 psr 规范，拿来即用
webprofil/crawler
401 Downloads
Crawls Website and can generate backstop json
survos/crawler-bundle
3476 Downloads
Provides a way to create tests that crawl a site