Download the PHP package vipnytt/cleanparam-url-filter without Composer
On this page you can find all versions of the php package vipnytt/cleanparam-url-filter. It is possible to download/install these versions without Composer. Possible dependencies are resolved automatically.
Download vipnytt/cleanparam-url-filter
More information about vipnytt/cleanparam-url-filter
Files in vipnytt/cleanparam-url-filter
Package cleanparam-url-filter
Short Description Clean-Param URL filtering class - Find and remove any duplicate URL
License MIT
Homepage https://github.com/VIPnytt/CleanParam-URL-filter
Informations about the package cleanparam-url-filter
Clean-Param URL filtering class
PHP class to filter URL duplicates, with integrated support for Yandex Clean-Param specifications.
FAQ:
What does it do?
It filters your URL lists so that any duplicate pages are removed.
What to expect if I'll filter my URLs?
- You'll never have to reload duplicate information again.
- More efficient web crawling.
- Server load will decrease.
What is Clean-Param?
It's a robots.txt directive witch describes dynamic parameters that do not affect the page content (e.g. identifiers of sessions, users, referrers etc.). When added, it has an significant impact on the number of URLs considered as duplicates. Learn more.
Installation
The library is available for install via Composer package. To install via Composer, please add the requirement to your ` file, like this:
and then use composer to load the lib:
You can find out more about Composer here: https://getcomposer.org/
Usage:
`` Pro tip: If you're going to filter tens of thousands of URLs, (or even more), it is recommended to break down the list to a bare minimum. This can be done by grouping the URLs by domain (or even host), and then filter each group individually. This is for the sake of performance!
Problem solving:
Fatal error: Maximum execution time exceeded.
Reason: You're probably trying to filter thousands of URLs.
- It is recommended to break down the list of URLs to a bare minimum. This can be done by grouping the URLs by domain (or even host), and then filter each group individually.
- Increase PHPs max execution time limit by using `. When called, it sets the time limit to 60 seconds, and restarts the timeout counter from zero.
- If you're already looping thou groups of URLs (like suggested), put ` into the loop, so that each time a new set of URLs is parsed, the timeout counter is restarted.
Fatal error: Allowed memory size of 134217728 bytes exhausted.
Reason: You're probably trying to filter tens of thousands of URLs, maybe even more.
- At this point, you're required to break down the list of URLs to a bare minimum. This can be done by grouping the URLs by domain (or even better, host), and then filter each group individually.
- Increase PHPs memory limit. This could be done by setting
or by changing the
variable in your `` file.