Download the PHP package paslandau/guzzle-rotating-proxy-subscriber without Composer

DEPRECATED ⛔

This repository has been deprecated as of 2019-01-27. That code was written a long time ago and has been unmaintained for several years. Thus, repository will now be archived.If you are interested in taking over ownership, feel free to contact me.

guzzle-rotating-proxy-subscriber

Plugin for Guzzle 5 to automatically choose a random element from a set of proxies on each request.

Description

This plugin takes a set of proxies and uses them randomly on every request, which might come in handy if you need to avoid getting IP-blocked due to (too) strict limitations.

Key features

• switches proxies randomly on each request
• each proxy can get a random timeout after each request
• each proxy can have a list of attached "identities" (an entity including cookies, a user agent and default request headers)
• a request can be evaluated via user-defined closure
• builder class for easy usage
• unit tests

Basic Usage

Examples

See examples/demo*.php files.

Requirements

• PHP >= 5.5
• Guzzle >= 5.3.0

Installation

The recommended way to install guzzle-rotating-proxy-subscriber is through Composer.

curl -sS https://getcomposer.org/installer | php

Next, update your project's composer.json file to include GuzzleRotatingProxySubscriber:

{
    "repositories": [ { "type": "composer", "url": "http://packages.myseosolution.de/"} ],
    "minimum-stability": "dev",
    "require": {
         "paslandau/guzzle-rotating-proxy-subscriber": "dev-master"
    }
    "config": {
        "secure-http": false
    }
}

Caution: You need to explicitly set "secure-http": false in order to access http://packages.myseosolution.de/ as repository. This change is required because composer changed the default setting for secure-http to true at the end of february 2016.

After installing, you need to require Composer's autoloader:

General workflow and customization options

The guzzle-rotating-proxy-subscriber uses the RotatingProxy class to represent a single proxy. A set of proxies is managed by a ProxyRotator, that takes care of the rotation on every request by hooking into the before event and changing the 'proxy' request option of a request. You might choose to further customize the request by adding a specific user agent, a cookie session or a some other request headers. In that case you'll need to use the RotatingIdentityProxy class.

The response of the request is evaluated either in the complete event or in the error event of the guzzle event lifecycle. The evaluation is done by using a closure that might be defined for each RotatingProxy individually. The closure gets the corresponding event (CompleteEvent or ErrorEvent) and needs to return either true or false in order to decide wether the request was successful or not.

An unsucessful request will increase the number of failed requests for a proxy. A distinction is made between the total number of failed requests and the number of requests that failed consecutively, because you usually want to mark a proxy as "unusable" after it failed like 5 times in a row. The number of requests that failed consecutively is reset to zero after each successful request.

You might define a random timeout that the proxy must wait after each request before it can be used again.

If all provided proxies become unsuable, you might either choose to continue without using any proxies (= making direct requests, thus revealing your own IP) or to let the process terminate by throwing a NoProxiesLeftException instead of making the remaining requests.

Mark a proxy as blocked

A system might block a proxy / IP due to a too aggressive request behaviour. Depending on the system, you might receive a corresponding reponse, e.g. a certain status code (Twitter uses 429) or maybe just a text message saying something like "Sorry, you're blocked".

In that case, you don't want to use the proxy in question any longer and should call its block() method. See next section for an example.

Use a custom evaluation function for requests

Since the "evaluation" is usually very domain-specific, chances are high that you have something already in place to determine success/failure/blocked states in your application. In that case you sohuldn't duplicate that code/method but instead use the GUZZLE_CONFIG_* constants defined in the RotatingProxyInterface to store the result of that method in the config of the guzzle request and just evaluate that config value. See the following example for clarification:

Set a maximum number of fails (total/consecutive)

Set a random timeout for each proxy before reuse

The first request using this proxy will be made without delay. Before the second request can be made with this proxy, a random time between 1 and 5 seconds is chosen that must pass. This time changes after each request, so the first waiting time might be 2 seconds, the second one might be 5 seconds, etc. The ProxyRotator will try to find another proxy that does not have a time restriction. If none can be found, a WaitingEvent is emitted that contains the proxy with the lowest timeout. You might choose to either skip the waiting time or to let the process sleep until the waiting time is over and a proxy will be available:

Define if the requests should be stopped if all proxies are unusable

If set to true, the ProxyRotator will not throw an NoProxiesLeftException if all proxies are unusable but instead make the remaining requests without using any proxies. In that case, a UseOwnIpEvent is emitted every time before a request takes place:

Use the builder class

The majority of the time it is not necessary to set individual options for every proxy, because you're usually sending requests to the same system (maybe even the same URL), so the evaluation function should be the same for every RotatingProxy, for instance. In that case, the Build class might come in handy, as it guides you through the process by using a fluent interface in combination with a variant of the builder pattern.

This would be equivalent to:

Use different "identities" to add customization to the requests

There are more advanced systems that do not only check the IP address but take also other "patterns" into account when identifying unusual request behaviour (that usually ends in blocking that "pattern"). To prevent being caught by such a system, the RotatingIdentityProxy was introduced. Think of it as a RotatingProxy with some customizations flavour to diversify your request footprint.

The customization options are handled via the Identity class and - for now - include:

• user agent
• default request headers
• cookie session
• use of the "referer" header

Note: Since RotatingIdentityProxy inherits from RotatingProxy it has the same capabilities in terms of random waiting times.

Randomly rotate through multiple identities

The RotatingIdentityProxy expects not only one identity but and array of identities. You can further provide a RandomCounterInterval the will randomly switch the identity after a certain amount of requests. From the outside (= the server receiving the requests) this looks like a genuine network of different People sharing the same IP address.

Use builder with identities

There are two options that can be used via the builder interface:

• distributeIdentitiesAmongProxies($identities)
• eachProxySwitchesIdentityAfterRequests($min,$max)

Frequently searched questions

• How can I randomly choose a proxy for each request in Guzzle?
• How can I avoid getting IP blocked?