Download the PHP package randak/charlotte without Composer
On this page you can find all versions of the php package randak/charlotte. It is possible to download/install these versions without Composer. Possible dependencies are resolved automatically.
Download randak/charlotte
More information about randak/charlotte
Files in randak/charlotte
Package charlotte
Short Description Charlotte crawls through your website and captures data in a database, including links between pages, H1 elements, scripts, stylesheets, and keywords.
License GPL-3.0
Informations about the package charlotte
Charlotte
Author: Kristian Randall [email protected] Copyright 2014
PHP-based web crawler for site analysis. Crawls your website and stores information about your pages, scripts and stylesheets in a Neo4j graph database. (Can be extended to use any database.)
Installation
Install using Composer:
composer require randak/charlotte:dev-master
Depending on your composer settings, you may need to run composer require everyman/neo4jphp:dev-master
before you can install Charlotte. If you get an error about that package not being
available, this is the likely solution.
In addition to installing Charlotte, you'll also need Neo4j, whether it be on the same machine or another server.
Configuration
After installation, you will need to set up your configuration. Currently, there
is an example config file in the examples
folder. The config will look something
like this:
You should set the URL here to be the homepage of the website you wish to crawl.
The exclude patterns are regular expressions that will match URLs you don't want to crawl. For example, we are ignoring certain file types, and any URL that starts with a #.
Usage
Charlotte is currently designed to be run from the command line only.
Create a file called crawl.php
.
touch crawl.php
Insert the follow.
Make sure everything is set up in your config.yml
and that your database is open.
Run the script.
php crawl.php
Contributions
Contributions are welcome! If you'd like to contribute, please create an issue first.
Disclaimer
This crawler is intended to be used only on websites which are owned or operated by you. The developer(s) of this tool are not responsible for any use of this code that violates any laws or otherwise causes harm, and are not liable for any misuse.
All versions of charlotte with dependencies
everyman/neo4jphp Version *
fabpot/goutte Version *
symfony/yaml Version 2.*