Download the PHP package parseword/massfetcher without Composer

On this page you can find all versions of the php package parseword/massfetcher. It is possible to download/install these versions without Composer. Possible dependencies are resolved automatically.

FAQ

After the download, you have to make one include require_once('vendor/autoload.php');. After that you have to import the classes with use statements.

Example:
If you use only one package a project is not needed. But if you use more then one package, without a project it is not possible to import the classes with use statements.

In general, it is recommended to use always a project to download your libraries. In an application normally there is more than one library needed.
Some PHP packages are not free to download and because of that hosted in private repositories. In this case some credentials are needed to access such packages. Please use the auth.json textarea to insert credentials, if a package is coming from a private repository. You can look here for more information.

  • Some hosting areas are not accessible by a terminal or SSH. Then it is not possible to use Composer.
  • To use Composer is sometimes complicated. Especially for beginners.
  • Composer needs much resources. Sometimes they are not available on a simple webspace.
  • If you are using private repositories you don't need to share your credentials. You can set up everything on our site and then you provide a simple download link to your team member.
  • Simplify your Composer build process. Use our own command line tool to download the vendor folder as binary. This makes your build process faster and you don't need to expose your credentials for private repositories.
Please rate this library. Is it a good library?

Informations about the package massfetcher

MassFetcher

MassFetcher is a multithreaded HTTP GET request utility. Give it a path to request, and a giant list of domains to request it from. Retrieved files are saved to disk (subject to configuration parameters). You may find MassFetcher useful if you want to perform various types of web analysis:

MassFetcher will go get the data; doing something with it is up to you.

Requirements

Installation

Clone this repository to a new directory and then run composer install. This will pull in the dependency (a logger) and set up the autoloader.

Copy config.php-dist to config.php.

Usage

Configure your settings inside config.php. Here you can set the target URI path you want to request, along with a bunch of options to modify MassFetcher's behavior. The options are explained in the comments.

Supply your list of target hosts in a file called domains.txt. The Alexa Top 1M list may come in handy, but do some small test runs first!

Run php fetcher.php to execute MassFetcher.

Retrieved files will be saved to a directory (defaults to data) in a series of hierarchical subdirectories.

The repository ships with a sample domains.txt containing 100 hostnames, a a config that will request /ads.txt from all of them, and the logger set to debug level. You should probably run once using these defaults, then examine the output.log file to see what's going on under the hood.

Resources and Performance

Performance will vary depending upon your hardware, internet connection, and configuration settings. Broadly speaking, with 64 threads I've averaged around 1,000 requests per minute from various commodity cloud instances.

MassFetcher may use significantly more bandwidth and disk space than you expect. Due to error pages, redirects, and oddly-configured servers, you're going to get plenty of junk data.

For instance, suppose you request /ads.txt:

Some of MassFetcher's settings can help mitigate junk data. In particular, the strict filename matching option will only write a fetched file to disk if the final destination URI, after all redirects, has the same base filename that you requested.

You should do some small test runs whenever you change configuration, before launching into an enormous fetch job.

All versions of massfetcher with dependencies

PHP Build Version
Package Version
Requires php Version >=7.1
parseword/logger Version >=1.0.2
Composer command for our command line client (download client) This client runs in each environment. You don't need a specific PHP version etc. The first 20 API calls are free. Standard composer command

The package parseword/massfetcher contains the following files

Loading the files please wait ....