Libraries tagged by crawl
adamb/sitemap
155 Downloads
Create a sitemap for your domain, crawls all URL's except those with nofollow links and those explicitly stated to ignore to create the sitemap
su-sws/nobots
43957 Downloads
This module blocks (well-behaved) search engine robots from crawling, indexing, or archiving your site by setting a "X-Robots-Tag: noindex,nofollow,noarchive" HTTP header.
johnhwhite/curl-robot
1586 Downloads
Parallel URL crawling extension to curl-easy.
crwlr/crwl-ext-browser
205 Downloads
Extension configurations for integration of crwlr/crawler-ext-browser into the crwl.io app.
crwlr/query-string
1764702 Downloads
A library for convenient handling of query strings used in HTTP requests.
crewlabs/unsplash
124276 Downloads
Wrapper to access the Unsplash API and photo library
crwlr/url
38058 Downloads
Swiss Army knife for URLs.
cravler/maxmind-geoip-bundle
536414 Downloads
Bundle integrating MaxMind GeoIP2 database into symfony application
crwlr/schema-org
6303 Downloads
Extract schema.org structured data from HTML documents.
crwlr/html-2-text
3712 Downloads
Convert HTML to formatted plain text.
cradlephp/handlebars
26018 Downloads
Ultra fast PHP7 implementation of handlebars.js
cradlephp/components
29488 Downloads
Common PHP7 Patterns and Constructs
cravler/faye-app-integration-plugin
39912 Downloads
cravler/faye-app-integration
37111 Downloads
cravler/faye-app-bundle
37579 Downloads