Download the PHP package webhubworks/site-crawler without Composer
On this page you can find all versions of the php package webhubworks/site-crawler. It is possible to download/install these versions without Composer. Possible dependencies are resolved automatically.
Table of contents
Download webhubworks/site-crawler
More information about webhubworks/site-crawler
Files in webhubworks/site-crawler
Download webhubworks/site-crawler
More information about webhubworks/site-crawler
Files in webhubworks/site-crawler
Vendor webhubworks
Package site-crawler
Short Description A straightforward site crawler
License MIT
Homepage https://webhub.de
Package site-crawler
Short Description A straightforward site crawler
License MIT
Homepage https://webhub.de
Please rate this library. Is it a good library?
Informations about the package site-crawler
Site Crawler
Use this site crawler as a quick way to crawl any website. This is useful to detect any slow pages or pages with HTTP errors.
Please use this crawler responsibly. Do not use it to crawl websites that you do not own or have permission to crawl.
Installation
composer global require webhubworks/site-crawler
Development
- To run the crawler locally (instead of using the globally installed version):
php site-crawler app:crawl URL
- To build the standalone app, run
php site-crawler app:build site-crawler
and specify the next version.
Usage
Use the help: site-crawler --help
Example: site-crawler https://example.com --limit=50 --basic-auth=user:pass --exclude=action,imprint
Roadmap
- [ ] Add support for websites containing links in JS generated markup
- [ ] Run requests in parallel
All versions of site-crawler with dependencies
PHP Build Version
Package Version
Requires
php Version
^8.2.0
ext-dom Version *
illuminate/http Version ^11.29
illuminate/validation Version ^11.29
spatie/url Version ^2.4
ext-dom Version *
illuminate/http Version ^11.29
illuminate/validation Version ^11.29
spatie/url Version ^2.4
The package webhubworks/site-crawler contains the following files
Loading the files please wait ....