Download the PHP package ilmlv/proxy-scraper without Composer

On this page you can find all versions of the php package ilmlv/proxy-scraper. It is possible to download/install these versions without Composer. Possible dependencies are resolved automatically.

FAQ

After the download, you have to make one include require_once('vendor/autoload.php');. After that you have to import the classes with use statements.

Example:
If you use only one package a project is not needed. But if you use more then one package, without a project it is not possible to import the classes with use statements.

In general, it is recommended to use always a project to download your libraries. In an application normally there is more than one library needed.
Some PHP packages are not free to download and because of that hosted in private repositories. In this case some credentials are needed to access such packages. Please use the auth.json textarea to insert credentials, if a package is coming from a private repository. You can look here for more information.

  • Some hosting areas are not accessible by a terminal or SSH. Then it is not possible to use Composer.
  • To use Composer is sometimes complicated. Especially for beginners.
  • Composer needs much resources. Sometimes they are not available on a simple webspace.
  • If you are using private repositories you don't need to share your credentials. You can set up everything on our site and then you provide a simple download link to your team member.
  • Simplify your Composer build process. Use our own command line tool to download the vendor folder as binary. This makes your build process faster and you don't need to expose your credentials for private repositories.
Please rate this library. Is it a good library?

Example code of ilmlv/proxy-scraper

Informations about the package proxy-scraper

CI Latest version Total downloads PHP version

Proxy Scraper and Validator

Scrape free proxy lists from 36 live, tested sources and individually validate proxy capabilities — anonymity level, latency, HTTP(S) method support and more. Supports http / https / socks4 / socks5 proxies.

[!WARNING] Free public proxies are highly not recommended for sensitive data transfer.

Why this library

Contents

Installation

Recommended installation method is via composer:

Usage

The snippets below assume Composer's autoloader is already loaded (require __DIR__ . '/vendor/autoload.php';). dump() comes from symfony/var-dumper — swap it for print_r()/var_dump() if you prefer.

Scrape all sources

Get a deduplicated set

Sources overlap heavily, so the same proxy is often returned by several of them. get() returns every occurrence (and is what stats() counts); unique() returns the same proxies flattened across all sources with exact duplicates removed. Two proxies are equal when their string form (protocol://[user:pass@]host:port) matches, so the same endpoint reached over a different protocol is kept.

Scrape a single source

Scrape only sources that are due

Each source declares a cron SCHEDULE; scheduled() runs just the ones due right now — handy for a frequently-polling cron job that should not hammer every provider on every tick.

Inspect per-source statistics

Configure a source

Extra config keys are appended to the source URL as query parameters, so you can tune sources that accept them (e.g. pubproxy.com). You can also supply your own Symfony HTTP client.

Handle scraper errors

A source that fails (network error, bad response, misconfiguration) does not throw — the exception is captured and exposed via errors(), keyed by the source class.

Proxy scraper sources

Currently implemented proxy sources:

Feel free to request more sources.

Proxy scrapers

Keep in mind that there is prepared multiple types of scraping libraries that can be used to simplify creation of your own source scrapers. Currently supported source data types:

Each of these accepts a $protocols map (['http' => $urlA, 'socks5' => $urlB, …]) when a provider publishes one endpoint per protocol: every URL is fetched and parsed with its key as the forced protocol, and a dead endpoint is skipped rather than aborting the source. Leave it empty to use the single $url.

Define a custom source

Extend one of the scraper base types (each already implements ScraperInterface via ProxyScraper), point $url at the resource, and hand the class to only()/add() — there is no need to register it in LoadProxies:

Bundled sources live in src/Sources/, one class per provider, named <Provider>[Variant] after the provider's identity — its domain or GitHub owner/repo (e.g. FreeProxyListNet, ShiftyTR), with a Variant suffix only when one provider exposes several distinct lists (e.g. FreeProxyListNetUkProxy). A source sets $url (or a $protocols map for a provider publishing one list per protocol), an optional $protocol, the format-specific knobs of its scraper base (see each Scrapers/* class docblock), and a SCHEDULE. Unlike a custom source, a bundled one is registered in LoadProxies::$scrapers so it runs under all()/scheduled().

Proxy validation

This library can also be used for proxy capability validation:

Validate a proxy

ProxyValidation accepts either a proxy string or a Proxy entity. Build it with make(), apply any optional configuration through the set*() methods (setDomainValidators(), …), then call run() to execute the checks; run() populates and returns the validation object. Construction itself performs no I/O.

run() returns the populated ProxyValidation; read each capability off its properties:

Property Type Meaning
valid bool Overall validity (false if a check threw — see error).
error ?ResponseError Set when the run aborted.
anonymityLevel ?string elite / anonymous / exposed.
ip ?IpValidation Whether the egress IP matches the proxy host, plus its country / organisation.
http ?MethodsValidation Forward HTTP support (absolute-form requests). For a SOCKS proxy this is its tunnel — SOCKS has no forward mode.
httpTunnel ?MethodsValidation CONNECT-tunnel-to-:80 support — how a chained proxy / forward-proxy gateway reaches the exit. Run for HTTP proxies only (SOCKS are already covered by http); null otherwise.
https ?MethodsValidation HTTPS support with certificate verification (CONNECT tunnel to :443). Fails on an expired, self-signed, hostname-mismatched, or intercepted cert.
httpsInsecure ?MethodsValidation The same probe with verification disabled, run only when https fails. Passing means the proxy tunnels TLS behind an untrusted cert; null when https passed.
domains ?DomainsValidation Opt-in per-domain reachability (see below).
ipVersion ?IpVersionValidation IPv4 / IPv6 egress capability.
validatedAt DateTimeInterface When the run executed.

Each MethodsValidation exposes a per-method result plus an average latency:

Custom domain validators

Domain reachability checks are opt-in — none run unless you ask for them. A validator extends AbstractDomainValidation and only declares what is specific to it: the domain NAME (used as the result key), the URL to request (and an optional non-GET METHOD), plus a validate() that decides whether the response is what the site should return. The bundled ExampleCom is the template:

Register the validator classes you want to run with setDomainValidators():

Scrape and validate

Validate every proxy a source returns:

The validation result looks like (the domains block lists only the validators you opted into — it is empty when none are configured). Every individual check also carries an error field — null when it succeeded, a {message, file, line} object when it failed (line is an integer):

Testing

The library ships with a PHPUnit test suite split into two suites:

Continuous integration runs on GitHub Actions (see .github/workflows/ci.yml): the unit suite runs across PHP 8.1–8.5 with code-coverage reporting, PHPStan static analysis runs as a separate gating job, and the live suite runs as a separate, non-blocking job on a weekly schedule (and on demand) so dead or drifting providers are caught early.

TODO:

All versions of proxy-scraper with dependencies

PHP Build Version
Package Version
Requires php Version ^8.1
ext-json Version *
ext-dom Version *
ext-simplexml Version *
dragonmantank/cron-expression Version ^3.3
symfony/http-client Version ^6.1
symfony/dom-crawler Version ^6.1
symfony/css-selector Version ^6.2
Composer command for our command line client (download client) This client runs in each environment. You don't need a specific PHP version etc. The first 20 API calls are free. Standard composer command

The package ilmlv/proxy-scraper contains the following files

Loading the files please wait ...