Download the PHP package thewinterwind/arachnid without Composer
On this page you can find all versions of the php package thewinterwind/arachnid. It is possible to download/install these versions without Composer. Possible dependencies are resolved automatically.
Download thewinterwind/arachnid
More information about thewinterwind/arachnid
Files in thewinterwind/arachnid
Package arachnid
Short Description A crawler to find all unique internal pages on a given website
License MIT
Homepage http://github.com/codeguy/arachnid
Informations about the package arachnid
Arachnid Web Crawler
This library will crawl all unique internal links found on a given website up to a specified maximum page depth.
This library is based on the original blog post by Zeid Rashwani here:
http://zrashwani.com/simple-web-spider-php-goutte
Josh Lockhart adapted the original blog post's code (with permission) for Composer and Packagist and updated the syntax to conform with the PSR-2 coding standard.
How to Install
You can install this library with Composer. Drop this into your composer.json
manifest file:
{
"require": {
"zrashwani/arachnid": "dev-master"
}
}
Then run composer install
.
Getting Started
Here's a quick demo to crawl a website:
You can set crawler to visit only pages with specific criteria by specifying callback closure using filterLinks
method:
<?php
//filter links according to specific callback as closure
$links = $crawler->filterLinks(function($link){
//crawling only blog links
return (bool)preg_match('/.*\/blog.*$/u',$link);
})
->traverse()
->getLinks();
How to Contribute
- Fork this repository
- Create a new branch for each feature or improvement
- Apply your code changes along with corresponding unit test
- Send a pull request from each feature branch
It is very important to separate new features or improvements into separate feature branches, and to send a pull request for each branch. This allows me to review and pull in new features or improvements individually.
All pull requests must adhere to the PSR-2 standard.
System Requirements
- PHP 5.6.0+
Authors
- Josh Lockhart https://github.com/codeguy
- Zeid Rashwani http://zrashwani.com
License
MIT Public License